[qmtest] Resource behavior questions, and overall comments

[Mark Mitchell]

--On Friday, March 01, 2002 01:16:03 PM -0600 "WECaputo at thoughtworks.COM" 
<WECaputo at thoughtworks.COM> wrote:

First, allow me to apologize for the long delay between the time your
message was sent and this response.

> According to the docs, tests associated with a resource will have the
> following behavior:
>
> R1.setUp
> t_1.run
> t_2.run
> t_3.run
> ...
> t_n
> R1.cleanUp
>
> Presumably, if a second resource (R2) existed, and was associated with a
> different set of tests this resource would be called *after* the cleanup
> of R1:

That's not what the docs mean to say; it's assumed that resources, like
tests, are independent.  (This is important so that you can run them
in parallel, for example.)

> What we find we need for all but the simplest testing, is nesting and
> atomic resource execution, along with the option to have a resource
> isolate each test in a set:

I'm not sure if I understand what you mean by "isolate" in this context.

> R1.setUp
> R2.setUp
> test
> test
> test
> R2.cleanUp  <-- order is important
> R1.cleanUp

Yes, you should be able to do this.  In particular, R2 should be able
to list R1 as a prerequisite.  If that is not always the case (sometimes
you want R2 without R1) you can make a new resource (R3) that runs R1
and then R2.  Note that creating that new resource is something that a
custom database could do for you -- you don't actually need to do it
explicitly.

> R1.setUp
> test
> R1.cleanUp
> R1.setUp
> test
> R1.cleanUp
> R1.setUp
> test
> R1.cleanUp

That's not what's meant by resource, in our system.  The scheduler
is allowed to do things this way, even though it doesn't -- but
there's no way for a user to specify this, as opposed to say:

  R1.setup
  test
  test
  R1.cleanup
  R1.setup
  test
  R1.cleanup

The idea behind resources are that they are things that are shareable
across tests.  If the cleanup/setup is an intrinsic part of the test,
the test class itself should handle that.

In other words, resources are purely an optimization.  You could
*always* put the resource action into the test class.  But, if the
setup is expensive, then resources allow you to do it only once.
(The canonical example is database testing, where you need to populate
the database, and then want to run a bunch of queries.  You could set
up the database before each test, but it might take a long time, so
you just do it once, then run all the tests, then tear it down.)

> A bigger issue for us, is in regard to change.  The structure of the
> resource calling code in QMTest is very difficult to change because the
> ordering seems tied to the database class and the loading of the resource,
> which in turn is tied to the directed graph architecture of the engine
> itself. We really can't plan around a tool that is hard to modify.

I'm not sure if I understand this comment.  You perhaps find the source
code difficult to modify?  (That could be.)  Or, you find the very
structure and design hard to modify?  (That could be too -- but is less
surprising; we had to make some choices when we designed the tool.)

I think what you're finding is that resources simply aren't designed
for what you want -- they're designed for something else.  My guess
is that what you want is a custom test class (or classes) instead,
but it is, of course, also possible that there is yet some third
thing of which we have not yet conceived!

> This is complicated by the lack of tests for the engine itself. "Eating
> our own dogfood" has been a big part of the usefulness of the testing rigs
> we've written before, even if you don't build them test first. We really
> can't rely on an engine that does not have testing.

Yes, this is a valid criticism.  At this stage, we do not have the
resources to build as many tests as we would like, but we do hope to
do that in the future.

> Sorry for providing so much critical feedback, I hope this doesn't sound
> too down on you -- overall I would definitely give you guys a thumbs up,
> and if it weren't for those issues, and we would definitely be
> evangelizing this tool through the XP community as we in the XP and agile
> world could use a good AT framework. We will keep watching qmtest with
> interest.

Feedback is appreciated.  I hope that future releases of QMTest will
better meet your needs.

And, of course, we are happy to provide custom modifications and
enhancements if that is something in which you would be interested.

--
Mark Mitchell                   mark at codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com