[qmtest] The state of GUI

[Vladimir Prus]

On Thursday 07 April 2005 13:13, Stefan Seefeld wrote:
> Vladimir Prus wrote:
> > 1. Even for initial tests run, this is not good. One must be able to look
> > at failures and decide which are expected.
>
> Would the ability to generate an expectation without actually running a
> test do what you want, 

Well, I think I can do it with a GUI already :-/ Not sure though.

> > 2. If new tests are added, expectations must be again set manually. If
> > you just rerun tests and use the results, then everybody must reevaluate
> > all tests failures.
>
> Not if you distribute not only the tests but also the expectation files,
> which at least one developer needs to create by running the test or
> otherwise.
>
> I'm not sure I understand your concern here, as the first thing I do when
> writing a new test is somehow expressing what result I expect. It's just
> that QMTest until now requires you to run the test at least once to do
> that.

I'm just arguing with the statement that one can just use results of a prior 
test run as expected result, without any manual editing. Imagine that I have 
just one configuration and an expected results file -- that contains manually 
changed expectation. If I add new test, rerun tests, and replace the 
expectations file, I loose the manually set expectations already stored 
there.

> > 3. Each test result is for specific compiler version. I really don't want
> > to explicitly edit test results for each version of some broken compiler
> > -- I want to specify that some compiler is broken.
>
> At what granularity do you want to indicate the brokenness ? If it is not
> per test, how else ? 

test/compiler_vendor combination. Say, "this tests fails on all versions of 
yfc", as opposed to explicitly listing 15 versions of "yfc" where the test 
fails.

> (Why running the test suite on that compiler if 
> 'broken' is not per-test ?)
>
> > The above means that editing expectation is not so uncommon expectation.
> > I also manually edit them for my project at work, for the same reason.
> > And I suppose some people would be more comfortable editing text file, as
> > opposed to using GUI, so allowing to specify expectations in a text file
> > would be nice.
>
> I believe there is a xml-based result format that QMTest understands.

Yea, but if people decide that editing 15 files for 15 versions of "yfc" is 
not too nice (even with GUI), they should be able to do something. Like using 
custom file format.

- Volodya