drupal planet

project.drupal.org (the scratch site for project.module) is now running code which will automatically set issues to code needs work when they fail automated testing. In the past, problems with marking issues incorrectly have led to this functionality been switched off, so the code has been refactored in several places to make things much more reliable.

However, it could use a final test before it's deployed live on Drupal.org, which is where you come in.

The following is an irc chat from #drupal-dev with hunmonk where we tried this out. Essentially - you should log into project.drupal.org (it's an old d.o database, so your d.o login should work), post a patch or two, then play with the links to make them fail or pass at will, set status manually. The log explains it all in more depth. (note that because project.drupal.org is an old database, it still thinks 6.x is HEAD, so pretend it's 7.x and you'll be fine).

Drupal.org has scheduled downtime on Friday to put some of the other pieces necessary for this in place, this is something that anyone can help with with a few minutes spare, so it'd be great to have some more positive testing so we can switch it on as soon as possible.

<a href="http://testing.drupal.org" title="http://testing.drupal.org">http://testing.drupal.org</a> - which will automatically run every patch for Drupal 7 through over 7100 (and counting) assertions needs one final push of manual testing before we can get it running at full capacity and reducing the workload of those of us who contribute regularly to Drupal core.

Over the past few weeks, Jimmy Berry (boombatower), Chad Phillips (hunmonk) and amazon (Kieran Lal) have been working hard on the final steps required to bring the testing platform up to date and have it ready to handle an increased capacity for patch testing. It's currently applying patches to a clean install of Drupal HEAD, running all tests, and reporting back the results to Drupal.org

The final stage is to get the platform marking patches to 'Code needs work' when they don't apply. There are currently over 400 patches needing review against Drupal 7, and clearing the queue of old stale patches is essential to getting this down to a manageable level.

Before we can do that, we need humans to verify the results of the automated tests to ensure they're accurate. Even if you've never reviewed a core patch before, you can help with this.

Step one: check out Drupal HEAD from cvs - <a href="http://drupal.org/node/320" title="http://drupal.org/node/320">http://drupal.org/node/320</a> has instructions

Step two: pick a link from <a href="http://groups.drupal.org/node/16209" title="http://groups.drupal.org/node/16209">http://groups.drupal.org/node/16209</a> - passing patches are at the top, failing patches are at the bottom. Failing patches are easier to test.

Step three. If you've picked a 'failing' patch, look at the reason why it failed. If it 'Failed to apply' then run these steps.
cd drupal7
wget <a href="http://drupal.org/the/link/to/the.patch" title="http://drupal.org/the/link/to/the.patch">http://drupal.org/the/link/to/the.patch</a>
patch -p0 < the.patch

If you get a message about 'failed hunks' or the patch otherwise fails to apply,you can then update the groups.drupal.org wiki to confirm this.

For more information on testing and applying patches see <a href="http://drupal.org/patch" title="http://drupal.org/patch">http://drupal.org/patch</a> - to test patches which pass, you'll need a working install of Drupal 7, which takes a little longer to set up, but not too much.

No, I'm not <a href="http://www.palantir.net/blog/august-and-everything-after">joining Palantir</a> (well hopefully for dinner), but I'll be attending the Taxonomy Sprint organised by the Encyclopedia of Life at the Field Museum in Chicago this week.

Taxonomy was the reason I installed Drupal 4.5.x back in 2005 (I wanted to categorise articles by author AND subject and none of the previous 4-5 CMSes I'd used could do it, oh those heady days), and while it's a very powerful system, it hasn't received a great deal of love the past 4-5 years. So when you're running up to 2 million terms through it, it starts to creak.

I'm still trying to get caught up after Drupalcon Szeged and sitting in a terminal in Minneapolis in between flights, so my laundry list of things I'd like to change about the taxonomy module won't make it into this post, but I'll try to post updates throughout the week.

As of today, after <a href="http://webchick.net/itch-of-the-week/fix-testing-crisis">months</a> of work, all core tests pass.

This means you should be able to download HEAD, install it, enable simpletest module, run all tests from admin/build/testing and see zero failures. If you can't, post a bug report, please.

By no means does this mean HEAD is bug free - we have about <a href="http://coverage.cwgordon.com/coverage/html/2">65% code coverage</a> (much of that implicit rather than explicit), and we actually <a href="http://drupal.org/node/276267#comment-906262">removed</a> two valid tests where they'd been committed without a correlating patch fixing the bug itself. So this is just the beginning of automated testing in core, and where the work really ramps up to make it an indispensable development tool.

The massive, massive advantage to a 100% pass rate is it's going to become much, much easier to spot regressions when patches are committed to core. In fact if we use the tests properly, we'll be able to stop the vast majority of regressions from getting committed at all. Less regressions means shorter code freeze, means longer code thaw!!

At a minimum, running all tests with a patch applied should become standard practice when doing reviews, and you can always check <a href="http://testing.drupal.org/tests" title="http://testing.drupal.org/tests">http://testing.drupal.org/tests</a> to see what it thinks too. That means no patch gets to RTBC (or stays at needs review long) which breaks a test. Got it? ;)

At the moment, tests are only going to catch a percentage of regressions. While they'll never be a panacea to clicking around yourself, and can't tell you if the interface is ugly or the code style is terrible, or measure performance etc., it's a very good first line of defence. We need to ensure that core tests are as comprehensive and robust as possible, since while they're a good first line of defense, 64% coverage is reminiscent of the Maginot line.

To do this, three things need to happen:

1. Write <a href="http://szeged2008.drupalcon.org/program/sessions/testing-part-2-awesome-testing-party">lots of tests</a> to fill in the <a href="http://drupal.org/project/issues?projects=3060&versions=156281&components=tests&states=1,16,8,13,14,15,2,4&priorities=&categories=&users=">gaps in core coverage</a>.
2. <a href="http://groups.drupal.org/node/13990">Get testing.drupal.org working</a>, so it'll sweep the issue queue, test if patches apply, run all tests, and mark issues to code needs work if they fail so we don't have to (my wrists will thank it).
3. Write tests to accompany existing bug fixes and feature requests in the issue queue, to confirm they work properly, and help get them committed more expediently.

So if you haven't got started with the whole testing thing yet, there really is no excuse any more.

Since tests for Drupal 7 keep breaking unexpectedly, I started doing <a href="http://groups.drupal.org/node/11993">test snapshots</a> every few days to track progress. After the first two runs there's good and bad news - less tests fail each time, but they're often different ones. Getting to a 100% pass rate is a pre-requisite for encouraging more people to write tests and extending the Drupal 7 release cycle, so hopefully we'll get the remaining ones fixed up pretty soon.