[qmtest] Failed prerequisite is not failing a test

[Mark Mitchell]

Scott Lowrey wrote:

> Mark Mitchell wrote:
>
>>
>> Certainly, having the execution engine clean up resources more 
>> aggressively would be fine.  In fact, they could be cleaned up after 
>> every use -- although that would be suboptimal.  But, the situation 
>> you're describing is a little more complex, in that the cleanup is 
>> mandatory.  In other words, the later tests require a "blank slate".
>>
>> We certainly can't model this now, and it certainly seems like a 
>> reasonable thing to want.  So, the question is what the design space 
>> looks like.  I'm almost tempted to say that we ought to make the 
>> execution engine an extension class, and then let you override it...
>>
>
> Good idea.  The structure of that particular module is not to be taken 
> lightly, but...
>
> ...we've already had occasion to come up with a second, slightly 
> modified exec engine for our own use.  It's a bit of a hack.  One of 
> our testing requirements is to run what we call an "endurance test" 
> (others might call it a "burn-in") where the execution engine must 
> loop through a set of tests for a defined interval of time - usually 
> days or even weeks.  If the "endurance" option is specified, 
> cmdline._ExecuteRun calls an alternate execution engine which has the 
> necessary modifications.
>
> Just thought I'd mention.  Not sure about how to define the base class 
> for an extensible ExecutionEngine...

Me neither.  It may be tricky to come up with an interface that is as 
tidy as that provided for Tests and Resources, although it does seem 
doable.  For example, the fact that a resource cannot be set up until 
another resource has been cleaned up is clearly something that _RunTests 
could manage.

The other option is to try to find some way to describe these 
dependencies in a generic way that the current execution engine could be 
taught to handle.  In the abstract, that might be even better, but we 
need to figure out what the right formalism is for these dependencies.  
In other words, we need to determine set of things on which a 
test/resource can depend.  At the moment, it can depend on a resource 
being set up, or on another test.  Perhaps all that we need to do is to 
allow tests/resources to depend on a resource being cleaned up.  That 
would also inolve adding a method to Target so that the execution engine 
could demand that a Resource be cleaned up, but that is relatively 
easy.  Thoughts?

-- 
Mark Mitchell
CodeSourcery, LLC
(916) 791-8304
mark at codesourcery.com