Spend another 30 minutes to get testing.drupal.org running

project.drupal.org (the scratch site for project.module) is now running code which will automatically set issues to code needs work when they fail automated testing. In the past, problems with marking issues incorrectly have led to this functionality been switched off, so the code has been refactored in several places to make things much more reliable.

However, it could use a final test before it's deployed live on Drupal.org, which is where you come in.

The following is an irc chat from #drupal-dev with hunmonk where we tried this out. Essentially - you should log into project.drupal.org (it's an old d.o database, so your d.o login should work), post a patch or two, then play with the links to make them fail or pass at will, set status manually. The log explains it all in more depth. (note that because project.drupal.org is an old database, it still thinks 6.x is HEAD, so pretend it's 7.x and you'll be fine).

Drupal.org has scheduled downtime on Friday to put some of the other pieces necessary for this in place, this is something that anyone can help with with a few minutes spare, so it'd be great to have some more positive testing so we can switch it on as soon as possible.

<hunmonk_> catch: first, the filters i have on project.drupal.org for file testing
<hunmonk_> catch: _all_ posted files that end with a .patch or .diff extension are made eligible for testing
<catch> hunmonk_: mind if I c&p this onto planet? That's probably the easiest way.
<hunmonk_> catch: that's fine
<hunmonk_> catch: now, i've included links that allow you to manually submit test results for testable files
<hunmonk_> catch: so any time you submit a testable file, you'll get a user message at the top that will allow you to manually do test results, either pass or fail
<hunmonk_> catch: these links appear regardless of any of the filters based on project, issue status, and release tag
<catch> hunmonk_: so - go to an issue, click a link, sends file for testing (reports back when it's done) ?
<hunmonk_> catch: once you've clicked the link in the user message, the file test results will appear
<hunmonk_> catch: no, go to an issue, submit a testable file, click the user message that appears for testing
* catch decides he should look at project.drupal.org while we do this ;)
<hunmonk_> catch: once a testable file has a test result associated with it, then you'll get permanent links to resubmit the file -- they will appear below the file test results for the file
<hunmonk_> catch: now, regarding the project, issue status, and release tag filters
<hunmonk_> catch: in the real world, pift would never send a file for testing that didn't pass the filters
<hunmonk_> catch: so for consistency, you should probably make sure the issue's project, issue status, and release tag match the existing filters before you click that first user message link
* Moonshine|afk is now known as Moonshine
<hunmonk_> catch: after that, you can go ahead and set the metadata for project, issue status, and release tag to other values to see how the system reacts
<catch> hunmonk_: OK, I'm here: http://project.drupal.org/node/188692
<hunmonk_> catch: making sense so far?
<catch> hunmonk_: scratch that - here: http://project.drupal.org/node/188694
<hunmonk_> catch: ok, did you submit that patch?
<catch> So I submit a patch, at needs review or rtbc against d7 that fits the naming conventions (patch or diff)
<hunmonk_> catch: no, not against d7
<hunmonk_> catch: you need to do it against d6
<catch> hunmonk_: ohhh.
<hunmonk_> catch: the reason why is that on p.d.o, d6-dev is what's associated w/ the HEAD release tag
<hunmonk_> catch: and our release tag regex is ^HEAD$
<catch> hunmonk_: OK, starting again http://project.drupal.org/node/188695
<hunmonk_> catch: you with me so far?
<catch> hunmonk_: http://project.drupal.org/node/188695 <<< :) :) :)
<catch> hunmonk_: yep.
<hunmonk_> catch: very good
<hunmonk_> catch: and you see the links to resubmit, right?
<catch> hunmonk_: yep.
<hunmonk_> catch: i mean the persistent ones below the patch test... ok...
<hunmonk_> catch: now, the filters
<hunmonk_> catch: project = Drupal
<hunmonk_> catch: issue status = CNR/RTBC
<hunmonk_> catch: release = 6.x-dev (for p.d.o only)
<hunmonk_> catch: you should definitely switch those around in different ways to make sure nothing breaks
<hunmonk_> catch: something else that is handy
<catch> hunmonk_: doing that now.
<hunmonk_> hunmonk_: if you delete the last comment, then it reverts the metadata to the previous comment
<hunmonk_> er, catch ^^^
<hunmonk_> catch: another thing i might suggest is to submit a few patches, and save the links that do the manual patches. then use those in different orders to make sure that it doesn't falsely move the issue to CNW
<hunmonk_> s/manual patches/manual test results
<hunmonk_> catch: make sense?
<catch> hunmonk_: that last bit doesn't, no.
<catch> hunmonk_: haven't managed to break it yet :)
<hunmonk_> catch: ok, so let's say you submit a patch on an initial issue, and two more patches in followups on the issue
<hunmonk_> catch: each time you submit one, copy paste the link that would submit the manual test results for the patch
<hunmonk_> catch: ie, set it aside somewhere handy
<hunmonk_> catch: do that for each file you submit first
<hunmonk_> catch: then when you have all the patches posted, and all the links handy, start using them to submit test results in some random order
<catch> hunmonk_: ahh, so the links at the top actually count for followups if there's a followup with a patch?
<hunmonk_> catch: you're trying to simulate the idea that the test results for different files may come back at different times
<hunmonk_> catch: yes indeed they do
<catch> hunmonk_: right, now it makes sense.
* catch does that.
<hunmonk_> catch: and we need to simulate that effect to make sure no false CNW's happen :)
<hunmonk_> catch: the workflow for the links is a tad clunky, i know. it was my first try and it works :P
<hunmonk_> catch: maybe someday i'll move them right below the issue attachment for even the inital posts
<catch> hunmonk_: this is just for testing right? I don't care what it looks like if it means we switch this on in two days.
<hunmonk_> catch: we're not switching this thing on in two days, most likely
<hunmonk_> catch: the upgrade happens in two days
<hunmonk_> catch: we still have those other issues to address
<catch> hunmonk_: the regexp, that seems like it could be done by then though.
<hunmonk_> catch: but we're making progress
<hunmonk_> catch: yes
<hunmonk_> catch: but the other pift-specific issues need to be addressed to avoid problems
<hunmonk_> catch: hopefully i can work w/ boombatower over the next two days to get those resolved
<catch> hunmonk_: saving links, this also appears to work fine, will try with followup links just in case.
<hunmonk_> catch: the most important thing is to return results in the order the files were submitted, and to make sure the issue doesn't CNW until the _last_ patch test returns a fail
<catch> hunmonk_: those links aren't forcing a fail for followups.
<catch> hunmonk_: see http://project.drupal.org/node/188696
<hunmonk_> catch: the last patch isn't marked failed. so it shouldn't
<catch> hunmonk_: I tried to make the last patch fail though.
<hunmonk_> catch: you may have gotten the link wrong
<hunmonk_> catch: i'm pretty confident those links work for followups
<catch> hunmonk_: well it should be the links up top no?
<hunmonk_> catch: yep
<catch> hunmonk_: not working for me.
<hunmonk_> catch: http://project.drupal.org/node/188696#comment-619684
<catch> hunmonk_: otherwise, so far so good.
<hunmonk_> working fine
<catch> hmm.
* Michelle2 is now known as Michelle
<catch> hunmonk_: I don't get links on my followups or anything.
<hunmonk_> catch: it's probably safe to just automatically mark a patch as passed after you submit it, then you have the permanent links by the patch resutls
<hunmonk_> catch: one moment
<hunmonk_> catch: http://project.drupal.org/node/188696#comment-619684
<hunmonk_> catch: you now have four passes in a row there, w/ the fail links right below the test results for each one
<hunmonk_> catch: go nuts :)
<catch> hunmonk_: can't break it.
<catch> hunmonk_: tried a bunch of combinations so far, and there don't appear to be any glitches.
<hunmonk_> catch: damn i'm good :P
<catch> heh.
<catch> hunmonk_: so I can try to get more people to do this, but 1. it wasn't that successful last time, afaik only pwolanin helped. 2. doesn't seem to be much more to test.
<hunmonk_> catch: i dunno. i would sure like more testing. i'm sure there's an edge case we've missed