News:

Use the "Forum Search"
It may help you to find anything in the forum ;).

Anyone interested in a continuous build?

Started by ArthurDenture, September 03, 2013, 03:41:23 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

ArthurDenture

(not sure if this is the best subforum - feel free to move wherever)

I decided last weekend to learn how to use Jenkins, the continuous build system, and naturally, I tried it out by setting up a build for Simutrans. You can see the results here: http://ec2-54-242-171-11.compute-1.amazonaws.com/

There are two nifty things at that site:

All it really needs at this point is to send notifications of broken builds somewhere useful. This is all on a tiny VM on Amazon's free tier (free micro instance for one year), which I just signed up for, so this particular server can keep running for at least a year. :-)

Does this sound useful to people? It's not a substitute for the nightly builds, but I think it'll be a good way to detect build breakage faster. And with some more work it can do more sanity checks and verify more platforms. I'm happy to create accounts on it for anyone who wants to play with the configuration.

Ters

I've mentioned this before, but an existing weekly build somewhere was considered enough. It is relatively rare that there are build time errors, though. When there are, it's not unlikely to be compiler specific.

To really be effective, it should build GDI, SDL Windows, SDL Linux and SDL Mac with graphics, and Posix without graphics. Perhaps also Makeobj. The first two should be built by both GCC and Microsofts compiler. And it should build from the Subversion repository, as it is the most up to date.

kierongreen

It's a good start certainly though :) If more platforms and sanity checks can be added even better :)

As for breaking builds - well look at the number of commits after 6624 to correct the compile errors that introduced for example.

Dwachs

Could it be extended in such a way that it can be used to produce patched builds? I.e. submit patch based on some revision and get the complete zoo of builds generated?

Could this be used to auto-post a message in the bug-reports board in case the build fails?

Another idea: It would  be nice to detect if the signature of functions exported to the scripting language changed. To do so it would be necessary to:
- build a debug build
- start it with '-debug 2' and load any saved scenario, quit immediately
- compare the files ~/simutrans/squirrel_types.awk with the file script/api/squirrel_types.awk from the repository
- issue a warning in case the files are different.
Parsley, sage, rosemary, and maggikraut.

Ters

Quote from: Dwachs on September 03, 2013, 08:09:19 AM
Could it be extended in such a way that it can be used to produce patched builds? I.e. submit patch based on some revision and get the complete zoo of builds generated?

I've never heard Jenkins used that way, and I've never seen anything suggesting such a possibility in normal Jenkins. It's probably easier to do without Jenkins, as the purpose of Jenkins is automated and/or repeatable builds. What Jenkins mainly offers is polling of VCS repos and test report generation, although there are some unorthodox ways of using it.

Quote from: Dwachs on September 03, 2013, 08:09:19 AM
Could this be used to auto-post a message in the bug-reports board in case the build fails?

At least if someone writes some fancy script (using wget, curl or whatever) to send the correct sequence of HTTP requests to the forum. It is normal for Jenkins to send such warnings to bug tracking software like Jira. At work, I have Jenkins commenting on the Jira issue that the submitted change is related as to whether the change broke the build/tests or not.

Quote from: Dwachs on September 03, 2013, 08:09:19 AM
Another idea: It would  be nice to detect if the signature of functions exported to the scripting language changed. To do so it would be necessary to:
- build a debug build
- start it with '-debug 2' and load any saved scenario, quit immediately
- compare the files ~/simutrans/squirrel_types.awk with the file script/api/squirrel_types.awk from the repository
- issue a warning in case the files are different.

Sounds possible, but it would perhaps be better to have a build target that generates an executable whos only purpose in life is to generate that file and quit.

Max-Max

When it comes to errors I think it is a matter of minutes or hours before someone discover that there is a compiler error in the trunk. So far the responsible person has fixed it as soon it is surfaced. When it comes to warnings there have always been about 100 warnings or so, during the time I have been here, and no one has fixed them.

So, yes it is a grate tool, but what use will it be if no one feels the responsibility to fix them?
- My code doesn't have bugs. It develops random features...

ArthurDenture

Thanks for the interest and responses, everyone!

QuoteWhen it comes to errors I think it is a matter of minutes or hours before someone discover that there is a compiler error in the trunk. So far the responsible person has fixed it as soon it is surfaced.

People are great at fixing errors once someone notices the breakage, but in fact certain categories of breakage have stuck around on the tree for a while: one recent example was COLOUR_DEPTH=0 not compiling and another was the mac build crashing on startup. It'll be nice to get as many of these secondary configurations (after linux+SDL and windows+GDI) as possible built automatically.

QuoteWhen it comes to warnings there have always been about 100 warnings or so, during the time I have been here, and no one has fixed them.

So now if you do fix some warnings, you'll have the satisfaction of seeing the warnings graph go down =p.

QuoteCould it be extended in such a way that it can be used to produce patched builds? I.e. submit patch based on some revision and get the complete zoo of builds generated?

That would be super nice. Two ways to set that up come to mind:
- https://wiki.jenkins-ci.org/display/JENKINS/Patch+Parameter+Plugin allows you to upload a patch to apply before building
- The git integration allows you to specify a set of branches to monitor, so one can imagine pushing to a new branch in a shared github repo

Either way, this would be a separate scratch build so that the main build continues to reflect the main repo. Ters is right that this is not so straightforward with Jenkins... I think the more formal way to solve this is to use Jenkins along with a git-based code review system like Gerrit, but I'm not going to try to convince everyone to adopt that. (Though I will give a +1 for code reviews, regardless of the interface used to do them.)

Getting the build to post to the forum would be neat. http://wiki.simplemachines.org/smf/SMF_API might be the way to go, or maybe a short Python script talking to the web interface will do the trick (if there's not javascript getting in the way, etc).

QuoteTo really be effective, it should build GDI, SDL Windows, SDL Linux and SDL Mac with graphics, and Posix without graphics. Perhaps also Makeobj. The first two should be built by both GCC and Microsofts compiler.

Makeobj should be straightforward. SDL Linux and Posix without graphics are covered. I've got Windows building on my Linux laptop using Mingw (which reminds me, I have a patch to send to fix that), so I probably can get that working on Jenkins. Mac is a little more challenging, as cross-compiling to darwin seems to be something of a dark art. I see two likely ways to make that work:
- The https://wiki.jenkins-ci.org/display/JENKINS/Build+Publisher+Plugin allows someone to use a physical Windows or Mac computer to upload build results to the server.
- The Lilypond project has written a crazy powerful cross compilation system, http://lilypond.org/gub/, which might work for us. It looks sufficiently generic.

And yes, getting warnings for the scenario api should be straightforward. (I agree with Ters that a separate binary to do this would be the cleanest solution.) I'll create an account for you and PM you its temporary password in case you want to try to set that up. (And if anyone else wants an account, let me know!)

Quoteit should build from the Subversion repository, as it is the most up to date.

Eh. The advantage of building from the Git repository is that Jenkins will link files and commits to Github, whereas I don't think the Simutrans svn server has a source browser. Plus I'd imagine a fair number of people use the Github repository as a basis for their work.

Ters

Quote from: ArthurDenture on September 04, 2013, 03:38:25 AM
Eh. The advantage of building from the Git repository is that Jenkins will link files and commits to Github, whereas I don't think the Simutrans svn server has a source browser. Plus I'd imagine a fair number of people use the Github repository as a basis for their work.

The synchronization from Subversion to Git can easily break down, as we saw this summer (or was it spring?). If the Subversion repository goes offline, like it also did this summer, nothing formally gets done anyway. It is also very important (at least to me, and likely also the core developers) that any error reports contain the Subversion revision number clearly (not just somewhere in small print, if at all). A Git hash means nothing to us.

As far as I know, Jenkins links builds to changelists/commits with every VCS. It doesn't need a source browser for that. Those interested in Jenkins CI results will have a local working directory and browse the files there, not online.

Simutrans also uses Subversion revision numbers as part of the version string for non-release builds, which unlike Git hashes is a simple counting number. A Jenkins build number can be used the same way, but unlike the global svn revision number, only compares within the same Jenkins job so one can't tell which is most recent of a Jenkins build and a non-Jenkins build. Bug reports for non-stable releases should also include Subversion revision number.

ArthurDenture

Two quick updates:

- I've added a build for makeobj. It compiled right out of the box, which is nice.
- I wrote a script using CasperJS (http://wwww.casperjs.org) that -- although I haven't tested it -- should log in to the Simutrans forum and post a message given on stdin to the bug reports forum. But I actually can't figure out an easy way to run that script in the same manner that Jenkins would send a failure email. Lack of that capability might explain why there are a billion different notification plugins that you'd think would just be shell scripts. Then again, I'm sure that some combination of plugins can make this work.

Ters

Everything Jenkins does could be a shell script, but shell scripts don't have a web interface with authetication, help buttons and status lights.

prissi

mail -S "subject" - "Hello" abc@simutrans.com

shoudl do the trick on Unix.

Ters


prissi


Ters


prissi

Maybe I misinterpreted
QuoteBut I actually can't figure out an easy way to run that script in the same manner that Jenkins would send a failure email.

Ters

No you didn't (or I do too). I simply missed that sentence, hence I wasn't aware of its contents.

ArthurDenture

I may have been unclear, but sending an email from the command line is not an obstacle. I know how to do that :-).

The issue is that I can't figure out how to configure Jenkins to run an arbitrary command in the case of a failed build, passing into that command the subject and body that it would have sent as a failure notification. This would be so I can pass that subject+body into a script that posts to the forum.

If the goal is just to have Jenkins send an email, that's already easily configurable in the web UI. But wiring it up to post to the bug report forum would be neat.

Ters

It's a bit surprising that noone has made a plugin for that, considering what there has been written plugins for. Maybe the problem is sending large pieces of text to a batch script?

Max-Max

Quote from: ArthurDenture on September 04, 2013, 03:38:25 AM
So now if you do fix some warnings, you'll have the satisfaction of seeing the warnings graph go down =p.
Believe me I have looked at some of them, but I'm not so familiar with that code so I didn't dare to fix them. Most of them are type auto conversion from a larger type to a smaller type. I don't know the impact of changing it to the larger type or if a explicit type conversion is allowed.
- My code doesn't have bugs. It develops random features...

Ters

One problem with some warnings is that if you try to silence them for GCC, you might end up with a warning, or even an error, on MSVC instead, and the other way around.

I have mostly given up fighting warnings at work, though I still try to avoid creating new ones myself. It's not easy to know how to avoid them. The list of todo comments also grows at a steady pace.

ArthurDenture

Quote from: Ters on September 08, 2013, 08:35:35 PM
One problem with some warnings is that if you try to silence them for GCC, you might end up with a warning, or even an error, on MSVC instead, and the other way around.

Not only that, even within gcc it depends heavily on what version you have and what glibc you're using. For instance, the continous build is clogged with warnings about ignoring the return value of chdir, which I haven't seen see on any other system. (Sure, it'd be nice to replace those calls to chdir with functions to build a full path to a file in the user dir or program dir, but the compiler warning is a bit much.)

QuoteThe list of todo comments also grows at a steady pace.

Tell me about it... at my work we have a script that counts TODO(whoever), and there are over 200 of them with my name.

isidoro

I have TODOs and TTOODDOOs, the latter the ones to be fixed before the next release...  and not a few of them...  ;)

Max-Max

(~sarcasm~) Hehe, have you guys not heard of a bug reporting system?!?  ;D

We finally got some order when we started to use Mantis BT (free PHP app)  :thumbsup:
- My code doesn't have bugs. It develops random features...

Ters

Quote from: Max-Max on September 09, 2013, 12:47:07 AM
(~sarcasm~) Hehe, have you guys not heard of a bug reporting system?!?  ;D

We finally got some order when we started to use Mantis BT (free PHP app)  :thumbsup:

The nice thing about them is that you can set an issue's tentative fix version to "sometime", assignee to nobody and due date to sometime in the year 9999. Saves clutter in the code.

prissi

It is even on sourceforge; but user tend to not report bugs there ...

isidoro

Quote from: Max-Max on September 09, 2013, 12:47:07 AM
(~sarcasm~) Hehe, have you guys not heard of a bug reporting system?!?  ;D

I'm rather old fashioned and use (should I say "had to use") a text-based one: http://www.ditrack.org/

ArthurDenture

I hacked at this some more last night and made the "make sure simutrans starts up" part of the build more effective. In particular, it used to just run simutrans in the background for 10 seconds and then kill it, in a way that ignored unsuccessful return codes. I've now fixed that:
- the sdl version is run with "-until 23341" so that it runs in fast-forward for a month and then exits cleanly. Nonzero return codes fail the build.
- the headless version doesn't do that because server mode doesn't fast forward, so instead it waits for 10 seconds and fails the build if the server didn't stay up

I also tried to get it running the SDL2 version, but that doesn't seem to want to start up under xvfb, or indeed in any configuration that lacks hardware acceleration. Haven't investigated that much.

prissi

You can run the server version on a socket, which it cannot bind to; it will then print an error message and exit.

ArthurDenture

I might not have been clear: right now for the server version it actually loads a game, runs it for 10 seconds (with zero clients), and kills it. So that's already better coverage than starting up with a bad port.

prissi

In ten seconds on my server it did not even finish to load the pakset. All together it took a minute for it to be ready.

ArthurDenture

With pak64 the server seems to start up pretty quickly on this relatively underpowered EC2 instance. You can see from the logs at http://ec2-54-242-171-11.compute-1.amazonaws.com/jenkins/job/simutrans-run-sdl/30/console that it does get to "Running world". (Note that some of the logs are out of order, not sure why. Buffering somewhere, I suppose.)

But yeah, 30 seconds will be safer, so I changed the configuration for future builds to wait 30s. Meanwhile, the primary build is still red right now due to the missing "svn add" (see bug report forum).

Max-Max

Can you pipe or read the logs from Simutrans and kill it 30 seconds after you have seen "Running world"? Is it possible?
- My code doesn't have bugs. It develops random features...

ArthurDenture

QuoteCan you pipe or read the logs from Simutrans and kill it 30 seconds after you have seen "Running world"? Is it possible?

Quite doable. The graphical versions are already run from within a Python script, since the screenshotting was getting a tad complicated for bash. You could imagine a similar script for the headless version that also checks stdout.

Even better would be to start one or two clients along with the server, adding coverage of at least the basic network commands that don't come from user behaviors.

I'm not sure when I'll have time to try that out, so I'll reiterate my offer for Jenkins admin rights and/or login access for anyone who wants it :-).