[qmtest] The state of GUI

[Craig, Dave]

Multiple results would be useful to get an idea of the reliability of a
test case.  A result may fail 100% of the time, 63% of the time, etc.  A
failure percentage seems to be a more reliable result when there are
poorly designed test cases.

Expectation based on a previous run will report changes for
intermittently passing test cases.  If the expectation were a little
richer, FAIL 100%, PASS 100%, and intermittent pass/fail, the reports
would reflect more significant changes.

Thanks,
     Dave

-----Original Message-----
From: Mark Mitchell [mailto:mark at codesourcery.com] 
Sent: Wednesday, April 06, 2005 9:21 AM
To: Vladimir Prus
Cc: qmtest at codesourcery.com
Subject: Re: [qmtest] The state of GUI

Vladimir Prus wrote:

> You mean "this test results are for configuration 1"? 

I'm not convinced that extending the results file format to contain 
multiple runs is that good of an idea.  The way I would expect the 2.95 
vs. 2.95+stlport vs. ... expectations would be handled is by simply 
running the testsuite is those configurations and saving the results 
files.  The GUI allows you to edit expectations, but the recommended way

of getting expectations set up is just running the testsuite.

Is that inconvenient?

-- 
Mark Mitchell
CodeSourcery, LLC
mark at codesourcery.com
(916) 791-8304