[qmtest] getting started

[Vladimir Prus]

John Schmitt wrote:

> > > - automatically create bugzilla tickets for
> > >   new/unexpected failures?
> >
> > No -- although adding automatic support for talking to defect tracking
> > systems is definitely something in which we are interested.
>
> In the case of bugzilla, this might be a place to start:
> http://quigley.durrow.com/bugzilla.html

This might be obvious, but still it's not mentioned. QMTest can output test 
results as XML file. It would be simple to read that XML, find tests with 
failed expectation, and open bugzilla issues for those, for example using
command line interface tool you mention. 

> > > - when reporting a failure, how can I specify which
> > >   tag should be used to check out the source when
> > >   attempting to reproduce the problem
> >
> > A failure in QMTest, or a failure in the tested sotware?
>
> A failure in tested software.  I guess the test suite needs to set some
> variable (the value of the tag) when the prerequisite is run (which
> checks out the software about to be tested)  and then reference that
> variable when the report is generated.  Advice on other approaches
> welcome.

I have not yet set up nightly builds/tests and there are no branches at the 
moment, but still.... when I get to it the the procedure will be simple

1. Checkout the needed version
2. Build
3. Test

I think this is more straigtforward than making resource for checking 
out/building. And less work: you don't have to make all your tests depend on 
the resource.

> I browsed the source a little and I think
> /home/qmtest/lib/qm/qm/test/file_database.py is what I was thinking
> about.  I fear that I don't quite understand the cost/benefit of
> plaintext/xml tradeoff here.  Am I missing some obvious benefit?

IMO, xml database is good because it's already implemented. I get the ability 
to manipulate all the fields from the gui and store tests. Other test formats 
might be good for special purposes (like testing compiler, where source file 
is natural test case)

- Volodya