Last week several of us attended the University of Baltimore to conduct usability testing of Drupal 7.
You should be able to see a video of our DrupalCon presentation soon at the DC 2009 site, and we'll be writing up more blog posts soon to give some background - but for those who can't wait to get stuck in, we've documented over 100 different issues at http://www.drupalusability.org/ - many of these now have drupal.org issues linked, some have patches ready to review already.
While we identified some major issues in testing, a lot of these are one line patches - so even if you've not done much core hacking before you'll be able to find something to work on. You can also help us by taking any issues from the site which don't have a corresponding drupal.org issue, and posting one up - we're tagging everything found in Baltimore with UBUserTesting2009.
After several wasted hours trying to work out exactly why nodes were so much slower than comments across several benchmarks, I've found out why...
anonymous users don't have the 'access comments' permission by default. Yes that's right, all the benchmarks were wrong. Like a race between one guy on sleeping pills and the other on steroids.
Nodes turn out to be almost identical to comments when comment module actually goes to the effort of showing them to anonymous users. Note these tests don't have Drupal's page cache enabled or anything like that. Having said that, given the disastrous results last time, I'd really appreciate some independent benchmarks - 10 nodes vs. 10 comments, 30 vs. 30, 300 vs. 300 - and Drupal 6 vs. Drupal 7 on those pages would be handy too. NB. Some cachegrind screenshots (aggregated from 30 requests each) are attached at the end of this post.
ab -c1 -n1000
/node with 90 nodes:
Server Software: Apache/2.2.8
Server Hostname: d7.7
Server Port: 80
Document Path: /
Document Length: 94593 bytes
Concurrency Level: 1
Time taken for tests: 369.436361 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 95179000 bytes
HTML transferred: 94593000 bytes
Requests per second: 2.71 [#/sec] (mean)
Time per request: 369.436 [ms] (mean)
Time per request: 369.436 [ms] (mean, across all concurrent requests)
Transfer rate: 251.59 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 344 368 18.1 364 514
Waiting: 337 360 17.8 355 506
Total: 344 368 18.1 364 514
/node/181 with 90 comments:
Server Software: Apache/2.2.8
Server Hostname: d7.7
Server Port: 80
Document Path: /node/181
Document Length: 227002 bytes
Concurrency Level: 1
Time taken for tests: 359.673295 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 227588000 bytes
HTML transferred: 227002000 bytes
Requests per second: 2.78 [#/sec] (mean)
Time per request: 359.673 [ms] (mean)
Time per request: 359.673 [ms] (mean, across all concurrent requests)
Transfer rate: 617.93 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 339 359 15.8 355 444
Waiting: 333 353 15.7 349 438
Total: 339 359 15.8 355 444
Percentage of the requests served within a certain time (ms)
50% 355
66% 360
75% 364
80% 368
90% 380
95% 394
98% 406
99% 418
100% 444 (longest request)
Time spent trying to correct the situation wasn't completely wasted, since I did some profiling of logged in page views and still found some interesting bottlenecks there, but still, a bit silly.
Good news though - there's no real arguments to have comments use their current hamstrung data storage any more.
About a year ago chx blogged about the necessary steps for comments to become nodes. Now we have both the code registry and multiple load, I thought it was worth seeing how far we've come.
Update:
It was a bit of a shock just how bad these benchmarks were, so I've since done some profiling. A lot of time is getting spent in drupal_render() (see http://drupal.org/node/353632 which includes a patch clawing something like 5-10% page generation time back too).
Update 2:
webchick reminded me about - http://groups.drupal.org/node/3550 - although the methodology/data is slightly different, we can see that database time has equalised, while those benchmarks also showed more time spent in PHP.
Update 3:
I've attached screenshots from kcachegrind, which show the most expensive functions for 30 nodes/30 comments and 90 nodes / 90 comments.
First, we need two relatively comparable pages to compare with varying numbers of nodes and comments - for this I've chosen node/n and taxonomy/term/n - I did a benchmark of these with one node and no comments on each to get a baseline. Since in both cases the node_load() is the same, we can compare the number of queries and requests per second for everything else.
node/n
Devel: Executed 32 queries in 9.58 milliseconds.
ab -c1 -n500
5.13 reqs/sec
5.28 reqs/sec
5.27 reqs/sec
taxonomy/term/n
Devel: Executed 33 queries in 11.45 milliseconds
ab -c1 -n500
5.21 reqs/sec
5.22 reqs/sec
5.28 reqs/sec
Close enough for our purposes that we don't need to make any adjustments.
project.drupal.org (the scratch site for project.module) is now running code which will automatically set issues to code needs work when they fail automated testing. In the past, problems with marking issues incorrectly have led to this functionality been switched off, so the code has been refactored in several places to make things much more reliable.
However, it could use a final test before it's deployed live on Drupal.org, which is where you come in.
The following is an irc chat from #drupal-dev with hunmonk where we tried this out. Essentially - you should log into project.drupal.org (it's an old d.o database, so your d.o login should work), post a patch or two, then play with the links to make them fail or pass at will, set status manually. The log explains it all in more depth. (note that because project.drupal.org is an old database, it still thinks 6.x is HEAD, so pretend it's 7.x and you'll be fine).
Drupal.org has scheduled downtime on Friday to put some of the other pieces necessary for this in place, this is something that anyone can help with with a few minutes spare, so it'd be great to have some more positive testing so we can switch it on as soon as possible.
http://testing.drupal.org - which will automatically run every patch for Drupal 7 through over 7100 (and counting) assertions needs one final push of manual testing before we can get it running at full capacity and reducing the workload of those of us who contribute regularly to Drupal core.
Over the past few weeks, Jimmy Berry (boombatower), Chad Phillips (hunmonk) and amazon (Kieran Lal) have been working hard on the final steps required to bring the testing platform up to date and have it ready to handle an increased capacity for patch testing. It's currently applying patches to a clean install of Drupal HEAD, running all tests, and reporting back the results to Drupal.org
The final stage is to get the platform marking patches to 'Code needs work' when they don't apply. There are currently over 400 patches needing review against Drupal 7, and clearing the queue of old stale patches is essential to getting this down to a manageable level.
Before we can do that, we need humans to verify the results of the automated tests to ensure they're accurate. Even if you've never reviewed a core patch before, you can help with this.
Step one: check out Drupal HEAD from cvs - http://drupal.org/node/320 has instructions
Step two: pick a link from http://groups.drupal.org/node/16209 - passing patches are at the top, failing patches are at the bottom. Failing patches are easier to test.
Step three. If you've picked a 'failing' patch, look at the reason why it failed. If it 'Failed to apply' then run these steps.
cd drupal7
wget http://drupal.org/the/link/to/the.patch
patch -p0 < the.patch
If you get a message about 'failed hunks' or the patch otherwise fails to apply,you can then update the groups.drupal.org wiki to confirm this.
For more information on testing and applying patches see http://drupal.org/patch - to test patches which pass, you'll need a working install of Drupal 7, which takes a little longer to set up, but not too much.
No, I'm not joining Palantir (well hopefully for dinner), but I'll be attending the Taxonomy Sprint organised by the Encyclopedia of Life at the Field Museum in Chicago this week.
Taxonomy was the reason I installed Drupal 4.5.x back in 2005 (I wanted to categorise articles by author AND subject and none of the previous 4-5 CMSes I'd used could do it, oh those heady days), and while it's a very powerful system, it hasn't received a great deal of love the past 4-5 years. So when you're running up to 2 million terms through it, it starts to creak.
I'm still trying to get caught up after Drupalcon Szeged and sitting in a terminal in Minneapolis in between flights, so my laundry list of things I'd like to change about the taxonomy module won't make it into this post, but I'll try to post updates throughout the week.
As of today, after <a href="http://webchick.net/itch-of-the-week/fix-testing-crisis">months</a> of work, all core tests pass.
This means you should be able to download HEAD, install it, enable simpletest module, run all tests from admin/build/testing and see zero failures. If you can't, post a bug report, please.
By no means does this mean HEAD is bug free - we have about <a href="http://coverage.cwgordon.com/coverage/html/2">65% code coverage</a> (much of that implicit rather than explicit), and we actually <a href="http://drupal.org/node/276267#comment-906262">removed</a> two valid tests where they'd been committed without a correlating patch fixing the bug itself. So this is just the beginning of automated testing in core, and where the work really ramps up to make it an indispensable development tool.
The massive, massive advantage to a 100% pass rate is it's going to become much, much easier to spot regressions when patches are committed to core. In fact if we use the tests properly, we'll be able to stop the vast majority of regressions from getting committed at all. Less regressions means shorter code freeze, means longer code thaw!!
At a minimum, running all tests with a patch applied should become standard practice when doing reviews, and you can always check http://testing.drupal.org/tests to see what it thinks too. That means no patch gets to RTBC (or stays at needs review long) which breaks a test. Got it? ;)
At the moment, tests are only going to catch a percentage of regressions. While they'll never be a panacea to clicking around yourself, and can't tell you if the interface is ugly or the code style is terrible, or measure performance etc., it's a very good first line of defence. We need to ensure that core tests are as comprehensive and robust as possible, since while they're a good first line of defense, 64% coverage is reminiscent of the Maginot line.
To do this, three things need to happen:
1. Write <a href="http://szeged2008.drupalcon.org/program/sessions/testing-part-2-awesome-testing-party">lots of tests</a> to fill in the <a href="http://drupal.org/project/issues?projects=3060&versions=156281&components=tests&states=1,16,8,13,14,15,2,4&priorities=&categories=&users=">gaps in core coverage</a>.
2. <a href="http://groups.drupal.org/node/13990">Get testing.drupal.org working</a>, so it'll sweep the issue queue, test if patches apply, run all tests, and mark issues to code needs work if they fail so we don't have to (my wrists will thank it).
3. Write tests to accompany existing bug fixes and feature requests in the issue queue, to confirm they work properly, and help get them committed more expediently.
So if you haven't got started with the whole testing thing yet, there really is no excuse any more.
Since tests for Drupal 7 keep breaking unexpectedly, I started doing <a href="http://groups.drupal.org/node/11993">test snapshots</a> every few days to track progress. After the first two runs there's good and bad news - less tests fail each time, but they're often different ones. Getting to a 100% pass rate is a pre-requisite for encouraging more people to write tests and extending the Drupal 7 release cycle, so hopefully we'll get the remaining ones fixed up pretty soon.
