[coldfire-gnu-discuss] Validation of gcc tools

[Mark Mitchell]

David Brown wrote:

> The question of development tools for various targets comes up regularly
> on comp.arch.embedded.  It can often lead to disagreements between users
> and supporters of gcc, and those who feel that only commercial closed
> source tools are good enough for professional use.  One of the key
> arguments from the closed-source camps is that they have tools that are
> validated and certified with Plum Hall and other such validation suites.

We (CodeSourcery) validate each and every release, using a combination
of open-source and closed-source testsuites.  These include tests of the
compiler, assembler, debugger, libraries, and other utilities.  Many of
the tests are in the open-source DejaGNU testsuite, which contains both
feature tests and regression tests.  As bugs are fixed in the tools, and
as new features are added, new tests are added to these testsuites to
ensure that they do not return.

We do also use Plum-Hall for validation.  GCC does not pass all tests in
the Plum-Hall C or C++ validation testsuite, but it does pass the vast
majority of them.  Many of the failures are failures to issue error
messages about technically invalid C or C++ code.  There are also
failures to accept valid corner-cases.  (For example, in C++, you can
overload based on whether not a particular function pointer type is
extern "C"; G++ does not implement that.)

We have automated tools that compare the hundreds of thousands of test
results produced during each release to the expected results, and
analyze any new failures, in general fixing those failures.

Part of the value of Sourcery G++ is that validation effort.  The GNU
development community (including CodeSourcery) works hard to validate
the tools, and part of the contribution process for GCC is running the
testsuite to ensure that changes do not cause the compiler to regress.

However, it is true that it is often the case that a change that works
well on one developer's test machine may cause problems on another.
And, some testsuites (like Plum-Hall) are only available to licensees
and therefore not accessible to all developers.  It is also true that
differences in configuration, build process, and so forth can critically
affect the reliability of the tools, even when building from the same
source code.  So, it is true that just downloading the GCC source code
and building it yourself results in a compiler that is not nearly as
well validated.

We do patch official FSF releases to fix defects revealed through our
testing.  (Those patches are then contributed back to the FSF for
inclusion in future versions of GCC.)

In summary, the GNU toolchain as a whole is validated in part as part of
the community development process.  In addition, we then do
significantly more testing, against the exact binaries we provide in our
both our zero-cost and commercial releases.

-- 
Mark Mitchell
CodeSourcery
mark at codesourcery.com
(650) 331-3385 x713