[qmtest] The state of GUI

[Stefan Seefeld]

Vladimir Prus wrote:

> 1. Even for initial tests run, this is not good. One must be able to look at 
> failures and decide which are expected.

Would the ability to generate an expectation without actually running a test
do what you want, especially if we add the possibility to annotate it ?

> 2. If new tests are added, expectations must be again set manually. If you 
> just rerun tests and use the results, then everybody must reevaluate all 
> tests failures.

Not if you distribute not only the tests but also the expectation files,
which at least one developer needs to create by running the test or otherwise.

I'm not sure I understand your concern here, as the first thing I do when writing
a new test is somehow expressing what result I expect. It's just that QMTest
until now requires you to run the test at least once to do that.

> 3. Each test result is for specific compiler version. I really don't want to 
> explicitly edit test results for each version of some broken compiler -- I 
> want to specify that some compiler is broken.

At what granularity do you want to indicate the brokenness ? If it is not
per test, how else ? (Why running the test suite on that compiler if 'broken'
is not per-test ?)

> The above means that editing expectation is not so uncommon expectation. I 
> also manually edit them for my project at work, for the same reason. And I 
> suppose some people would be more comfortable editing text file, as opposed 
> to using GUI, so allowing to specify expectations in a text file would be 
> nice.

I believe there is a xml-based result format that QMTest understands.

Regards,
		Stefan