[qmtest] Prerequisite Logic

[Nathaniel Smith]

On Mon, Jun 28, 2004 at 05:06:39PM -0400, Scott Lowrey wrote:
> When I select a single test to run, and that test depends on a 
> prerequisite test and a resource, why does QMTest run the resource but 
> not the prerequisite test?  I can understand skipping the prereq  if 
> QMTest already has a PASS status for it (web only), but this isn't the 
> case.  QMTest effectively ignores the prerequisite dependency when 
> "cherry picking" tests to run.

Yes, this is the intended behavior.  This is documented in the manual,
though perhaps not highlighted as much as it should be:
  http://www.codesourcery.com/public/qmtest/qm-2.2/manual.html/sec-qmtest-concepts.html#sec-tests
  http://www.codesourcery.com/public/qmtest/qm-2.2/manual.html/sec-running-tests.html#sec-ordering-and-dependencies

> I guess I can understand the logic that says, "You told me to run test X 
> and test X only."  But it seems a bit counterintuitive that QMTest can 
> treat prerequisites differently depending on how the user specified what 
> to run.

The idea is that tests and resources have different semantics.  A
Resource provides, well, a resource; a Test determines the truth value
of some statement about the thing being tested (either "X works"
or "X doesn't work", usually).  If I "depend on a Resource", that
means that my code won't execute properly without that Resource.  If I
"depend on a Test", that means that there's no point in running me
when that test fails, because we already know that whatever I'm
testing is broken.  Test dependencies are just an optimization, if you
want to think about it that way.

This distinction has other consequences too.  E.g. if you're running
across multiple machines, a given Resource may be set up multiple
times, once on each machine that needs it, whereas a test will never
be run more than once, no matter what.

> If the idea is to avoid running tests that the user didn't specifically 
> ask for, then maybe it would be best if QMTest returned an error (or 
> warning?) stating that the desired test cannot run until the 
> prerequisite does.  This would get pretty hairy with a command-line 
> invocation but would probably suffice for the web interface.

Basically, one needs to understand that QMTest isn't make, and that if
"the desired test cannot run until the prerequisite does" this is
something of a bug in your test suite.

I say "something" because it turns out there are sometimes cases when
this behavior is obnoxious, and one has to use kluges to work around
it.  Unfortunately, we don't have a very good solution for these cases
yet...

-- Nathaniel