[qmtest] writing composite tests

[Mark Mitchell]

On Mon, 2003-09-08 at 09:54, Stefan Seefeld wrote:
> hi there,
> 
> I have a couple of tests that consist of various steps
> such as compiling, linking, executing, result (output)
> analysis.
> 
> Each of these steps may be considered a test in itself,
> depending on the earlier steps. As such, I would model
> the situation as a 'composite test'.

We've actually got some pretty nice test/database classes for
compiling/linking/executing programs.  We should probably get those into
QMTest proper, but we've never had the time to check over the interfaces
as much as we'd like, and I'd like to avoid changing them after they go
into QMTest.

You can get them out of the "qtmc" module in the same repository as
QMTest.

> Can such an approach be supported by qmtest ?

It was a fundamental design decision for QMTest that every test would
have exactly one "outcome" (i.e., PASS/FAIL/UNTESTED/ERROR).

There are tree ways to model what you suggest within QMTest.

One is to use prerequisite tests; the linking test is only run if the
compiling test has already PASSed.

The second is to use a single test with multiple result annotations. 
The test passes only if everything works, but it if it fails there is an
annotation showing which step failed.

The third way is to combine the first approach with a "suite"; you have
tests "compile_a", "link_a", "run_a" which have one-another as
prerequisites, and then you have an "a" suite which encompasses all
three.  Then, you can say:

  qmtest run a

and the thing that you expect to happen will happen.

> success ? Then I could write my composite test's 'Run'
> method simply as 'a.Run() and b.Run()'.

We've intentionally made it hard for one Test to get its hands on
another Test.  The reason is that tests should be independent of
another; this is another design goal.  If tests are not independent of
one another then running one test is no longer a well-defined operation;
each test is potentially dependent upon context created by other tests.

-- 
Mark Mitchell
CodeSourcery, LLC
mark at codesourcery.com