The Karma Project: Code Less, Teach More

June 8, 2009

Automating Builds, Testing, and QA for E-Paath and (eventually) Karma

Filed under: News — bryanwb @ 7:39 am

I feel bad that lately I haven’t spent much time assisting Subzero w/ the coding of Karma. My current excuse is that I am working on a related if not immediate goal. Here in Nepal, we need to roll out a new build of the E-Paath learning activity. E-Paath is a huge activity , comprising more than 100 individual lessons. If I stick E-Paath into a single XO bundle, it is almost 1 GB in size. Testing E-Paath is also a major hassle since there are so many individual lessons and our eventual users are not particularly sophisticated. We have a fairly complex build process and we really need a straightforward testing and bundling process.

The holy grail for me is a web application that will tell me which tests failed in which build on which day. This sounds simple, but it is actually quite a complex problem.

We have to automate QA, regression testing, and reporting the results to a web page. Testing by hand is far to time-consuming. I believe that these problems will later affect organizations that use Karma to build large suites of lessons tuned to their locale and educational context. I do see a standard build and testing process to be important parts of Karma in the long-run. I really want developers with little knowledge of systematic testing, build processes, or QA to be able to re-use the hard work we are having to do here to get these processes together from scratch.

While unit-testing, QA, regression testing are in Karma’s future, they are in E-Paath’s immediate present. I have to get this stuff working ASAP for E-Paath.

E-Paath is different from most large software projects. The individual lessons are particularly dependent on each other. They are dependent on the “framework” that we use for a common layout for the lessons. Changes to the framework affect all of the lessons but changes to a particular lesson generally do not affect the other lessons.

I have tried to set up buildbot and found it quite complex to set up. It seems like a great tool to distribute running builds but that is not currently one of my requirements. I spent about a day w/ it last week before becoming quite frustrated. It seems like a great piece of software but its complex set up process implies to me that it will require a fair bit of maintenance.

Last Friday, I happened Martin Fowler’s essay on Continuous Integration. Really a good read. Soon after someone on the #buildbot IRC channel told me that Hudson may be a better fit for my particular needs. It is super-easy to test out hudson. Just download hudson and type in
sudo java -jar hudson.war

I have successfully created and run some test builds with hudson. So far it is quite easy. The next big step is to run regression and unit tests and output the results to a lovely chart like this:

Example of Test Results in Hudson
Warning — the above is an example from the Hudson demo. I haven’t gotten it running yet.

Putting the test results into the proper format is a non-trivial problem. You have to output your unit test results into a schema compatible with the JUnit test result format. I am a complete n00b when it comes to unit testing. I think I did a tutorial all of of 8 months ago. I have mucked around with pyunit and XMLTestRunner without much success for several hours this morning.

Now i have shifted my focus to nose which has a handy --with-xunit switch that could make outputting reports to Hudson a cinch! I will let you know my results.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: