[qmtest] Where to put 'import', function defs etc.

[Stefan Seefeld]

Rowan Sylvester-Bradley wrote:
> Hi,
>
> I'm trying to use QMtest for the first time and am having some 
> difficulties getting to grips with it. I hope someone can answer the 
> following questions for me (or tell me where to find a tutorial 
> explaining all this). I'm trying to test a remote device so all my 
> tests send messages via the serial port and receive serial responses. 
> I'm trying to use the python.ExecTest test class, and write all my 
> tests in Python.
>
> 1. Where do I put 'import' statements, function definitions etc. that 
> my tests need?

Your own test classes live in ordinary modules, so all the normal rules 
for coding in Python apply. What turns a test class into a 
QMTest-visible extension class is its registration via QMTest/classes.qmc.

> 2. If many of my tests need the same imports and functions, is there a 
> generic place to put the imports and function defs where they will be 
> accessible to all tests, or do I have to put them in every test?
I presume you put your test definitions into Python modules inside the 
QMTest/ directory in your test database root directory.
I believe that directory is being searched for for imports, so you could 
simply add arbitrary python modules (or entire packages) with shared 
code there, too.

>
> 3. It looks as if each test in QMtest must be self contained and 
> atomic, i.e. it must not depend on which tests have already been run, 
> or in which order.

That's right.

> If this is right, I don't understand how to deal with a situation that 
> needs a long sequence of messages to the device under test and 
> responses. The prerequisite concept doesn't seem to solve the problem, 
> because (as I understand it) although it guarantees that some tests 
> will have been run before the current test, it does not guarantee in 
> which order, or whether other tests will also have been run, all of 
> which will change the state of the device under test.
Right. As tests are (by design) self-contained, they must not result in 
any external state-changes affecting other tests.

If you need to prepare the environment the test will run in, you may put 
that into a 'resource' 
(http://www.codesourcery.com/public/qmtest/qmtest-snapshot/share/doc/qmtest/html/tutorial/concepts-resources.html)

Resources can be shared among many tests (and QMTest will make sure 
resources are set up prior to the dependent test being run, on the 
target the test is being run). In particular, this means a resource is 
set up only once. If you have a sequence of preparational steps that 
need to be repeated for each test, you have to execute them from within 
your test.


> If I genuinely have to make every test independent of what's gone 
> before, I will have to start each test with a reboot, and then add 
> commands to the test one by one - so if my test sequence consists of 
> reboot, send command A, receive response A, send command B, receive 
> response B, send command C, receive response C I will have to have 
> tests 1 (reboot, send A receive A), 2 (reboot, send A, receive A, send 
> B, receive B) and 3 (reboot, send A, receive A, send B, receive B, 
> send C, receive C). If I have 100 or more steps in my sequence, this 
> could be very tedious. If I could force my tests to run in sequence, I 
> could just have tests 1 (reboot, send A, receive A), 2 (send B, 
> receive B) and 3 (send C, receive C).

Why don't you put all the different steps into a single test ? Put 
appropriate validation checkpoints in between the steps, so your test 
lot will tell at which point things have broken, in case it fails.

(Compare that to a CompilationTest involving first compiling a source 
file, then running it. The multiple steps are 1) preprocessing, 2) 
compilation, 3) linking, 4) execution. In case of failure the test will 
report in which of those steps an error happened.)


> 4. How do I make the execution of a test dependent on the results of 
> previous tests (i.e. how do I access the results of previous tests 
> from within the Python code of a new test)?

You don't, as per the design that stipulates all tests being 
independent. (To illustrate one of the rationales: imagine running tests 
in parallel, possibly on a build farm: Test B may run in a different 
universe than test A, and both not knowing anything about each other.)

The concept of 'prerequisite tests' is simply meant as an optimization, 
so you can skip a test it is known to fail, because some other 
'prerequisite' test failed, too. See
http://www.codesourcery.com/public/qmtest/qmtest-snapshot/share/doc/qmtest/html/tutorial/concepts.html#concepts-tests

> 5. How do I make the sequence of tests dependent on the results of 
> previous tests? E.g. if test B passes, do test C and then test D, or 
> if it fails, do test E and then test F.

See above.

> 6. If I'm using the "Python Source Code" part of the python.ExecTest, 
> and not the Python Expression section, what is the recommended way of 
> indicating to QMtest whether the test has passed or failed?
As per the python.ExecTest's documentation, in the absence of an 
expression, the test fails if it throws an exception.

> 7. Should or must I make every comparison of expected results with 
> actual results a separate QMtest test, or can/should I have multiple 
> comparisons within a single QMtest test? In the latter case there 
> would obviously be multiple ways in which the test could fail, which 
> would need to be reported.
Right. Existing test classes are probably not flexible enough if you 
want to distinguish multiple such failure codes.
In that case you may want to write your own test classes and then log 
the failure details in the Result object.

> What are "resources"? Does using these help me solve any of my problems?

Possibly. See above.

> 8. If I want to create a new test class, where do I put the class 
> definition?
That's explained in 
http://www.codesourcery.com/public/qmtest/qmtest-snapshot/share/doc/qmtest/html/tutorial/extending.html

>
> 9. Is there a simple example of how to add a new test class written in 
> Python somewhere?

I'm not sure whether we have complete examples. However, the above 
documentation explains all the details. You may want to copy an existing 
module (such as qm/test/classes/python.py) into your QMTest/ directory, 
register the classes via QMTest/classes.qmc, and then experiment with 
modifications you apply to that code.

Good luck !

       Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...