[qmtest] Failed prerequisite is not failing a test

[Mark Mitchell]

Scott Lowrey wrote:

> Mark Mitchell wrote:
>
>>
>> This is a common confusion: if you do not explicitly list the 
>> prerequisite among the tests you want to run, then it will not be 
>> run.  We see prerequisites as purely an optimization: they provide a 
>> way of skipping some tests if earlier tests do not work the way you 
>> think.  It's possible that this was a bad design decision; it is 
>> certainly true that you are not the first person to have been 
>> confused.  However, it is how QMTest presently works.
>>
>
> Very well put.  :)  And very aggravating at times.  To me, a prereq is 
> a prereq whether I asked for it or not.  But maybe that's just 
> semantics.  Either way, I accept the current operation and, if I 
> *really need* to have something run prior to my test, then a resource 
> is probably a better alternative.
>
> Can I take this opportunity to segue into another issue?  We've 
> recently come across a situation where the resource behavior is 
> somewhat unacceptable in that the cleanup action occurs to late in the 
> session - i.e., when all tests have completed. 
>
> What do you think about having a resource clean up as soon as the 
> tests that depend on it have completed?  This goes way back to my 
> argument for a "suite resource" but with a slightly different spin.  
> The idea seems to  make perfect sense - we have situations where the 
> set-up tasks performed by a resource must be undone before proceeding 
> with other tests.   Any other ideas on how to do this?  If not, I can 
> try to make it work and submit the code.... assuming it's feasible.

Certainly, having the execution engine clean up resources more 
aggressively would be fine.  In fact, they could be cleaned up after 
every use -- although that would be suboptimal.  But, the situation 
you're describing is a little more complex, in that the cleanup is 
mandatory.  In other words, the later tests require a "blank slate".

We certainly can't model this now, and it certainly seems like a 
reasonable thing to want.  So, the question is what the design space 
looks like.  I'm almost tempted to say that we ought to make the 
execution engine an extension class, and then let you override it...

-- 
Mark Mitchell
CodeSourcery, LLC
(916) 791-8304
mark at codesourcery.com