[qmtest] The state of GUI

[Vladimir Prus]

On Wednesday 06 April 2005 20:20, Mark Mitchell wrote:
> Vladimir Prus wrote:
> > You mean "this test results are for configuration 1"?
>
> I'm not convinced that extending the results file format to contain
> multiple runs is that good of an idea.  The way I would expect the 2.95
> vs. 2.95+stlport vs. ... expectations would be handled is by simply
> running the testsuite is those configurations and saving the results
> files.  The GUI allows you to edit expectations, but the recommended way
> of getting expectations set up is just running the testsuite.
>
> Is that inconvenient?

Kind of. 

<aside>
I'm not saying that QMTest should take Boost requirements as the only target, 
and in fact I'm not sure many Boosters will be interested in using QMTest, 
and there are some deeper issues than just annotation format (Stefan should 
remember), but Boost is the good example for multi-platform project.
</aside>

Yes, it's easy to store results for each configuration and then use them as 
expectations. But I think we use different meaning of "expected" failure.
You take in as the failure which present in some previous run. I take it as a 
failure which was present in some previous run, and for which the developer 
has explicitly decided that this failure should not be fixed, and so for all 
subsequence run the failures should not specially announced.

For Boost, with 10 different compilers some of which are rather broken, this 
approach is a lot better then looking at all failures and trying to recall 
which must be fixed.

So:
1. Even for initial tests run, this is not good. One must be able to look at 
failures and decide which are expected.

2. If new tests are added, expectations must be again set manually. If you 
just rerun tests and use the results, then everybody must reevaluate all 
tests failures.

3. Each test result is for specific compiler version. I really don't want to 
explicitly edit test results for each version of some broken compiler -- I 
want to specify that some compiler is broken.

The above means that editing expectation is not so uncommon expectation. I 
also manually edit them for my project at work, for the same reason. And I 
suppose some people would be more comfortable editing text file, as opposed 
to using GUI, so allowing to specify expectations in a text file would be 
nice.

The point 3) means that it might be good to specify expectations independently 
from specific test results.

OTOH, it might be reasonable to start with simpler model of just a collection 
of expected result files.

- Volodya