are the LLVM pro's and cons of QMTest still valid?

[Mark Mitchell]

On 2/22/2011 1:51 AM, Andreas Saebjoernsen wrote:

> I assume the concerns from the LLVM mailing list are not valid, and that
> is good.

I don't claim to have reviewed all the traffic; just the summary you
posted.  Some of the cons were not accurate; I'm not sure if that's true
of all of them.

> 1. The last update to QMTest was in 2008. Has the current features of
> QMTest been  
>     verified to work correctly on newer and older versions of linux,
> windows and Mac OS X?

No.  We're using it on recent Ubuntu, but not on other systems.

>     Our project pass 'make check/distcheck/clean/distclean'. Most of our
> tests are functional 
>     tests that test specific features with a custom program. The
> tests depend on many test 
>     programs that are compiled with make. 

You would certainly have to do some refactoring.  You would end up with
"make check" running "qmtest ..." to actually run the tests, and you'd
want the tests themselves to be organized within QMTest.

>     of testing. One of my biggest questions is how to support libtool
> since we don't install libraries or 
>     programs before building and testing.

QMTest has the concept of a "resource" -- i.e., a set-up step that
occurs before some tests run.

> 3. How is the usability of QMTest when testing locally on a developers
> machine? 'make check' can 
>     easily run the tests for a directory at a time, and if required it
> will automatically rebuild dependencies.

Running "qmtest run" is approximately the same as running "make check"
in that respect.

The basic tradeoff is that you will spend more time setting things up
with qmtest in order to leverage its more-structured environment.

I would not characterize either DejaGNU or QMTest as "products" at this
point; they're useful technologies.  If you want to be successful with
either one, you might want to engage someone to help; otherwise, it's
going to be a fair amount of digging about on your own.

-- 
Mark Mitchell
CodeSourcery
mark at codesourcery.com
(650) 331-3385 x713