[qmtest] RFE: distinguish qmtest failure and test failures in qmtest's return value
Mark Mitchell
mark at codesourcery.com
Wed Aug 20 18:51:11 UTC 2003
On Tue, 2003-08-19 at 07:03, Stefan Seefeld wrote:
> hi there,
>
> I'm trying to execute 'qmtest run' from within a Makefile. Unfortunately
> with the current qmtest code base there is no way to detect whether
> qmtest's execution itself failed (say, because the 'qmtest' command
> couldn't be found) or whether the error stems from an (unexpected)
> failure of a test execution.
>
> qmtest should use different return codes to indicate the type of
> failure, such as '-1' for an internal error and any other non-zero
> number to indicate test failures. Then I can write my Makefile to
> just ignore failed tests as I deal with these failures on a different
> level.
Stefan --
I think this is a great idea.
I checked in the attached patch, which makes QMTest return 2 for failing
tests and 1 for all other errors. I've also documented this behavior;
until now the documentation made no promises about the return code.
Thanks,
--
Mark Mitchell <mark at codesourcery.com>
CodeSourcery, LLC
-------------- next part --------------
2003-08-20 Mark Mitchell <mark at codesourcery.com>
* NEWS: Mention documentation of exit status.
* qm/test/cmdline.py (QMTest.Execute): Use the return value from
__ExecuteCreateTdb.
(QMTest.__ExecuteCreate): Return a value explicitly.
(QMTest.__ExecuteCreateTdb): Likewise.
(QMTest.__ExecuteCreateTarget): Likewise.
(QMTest.__ExecuteSummarize): Likewise.
(QMTest.__ExecuteRun): Likewise.
* qm/test/qmtest.py: Simplify handling of exit codes.
* qm/test/classes/mount_database.py: Fix typo in doc-string.
* qm/test/doc/reference.xml: Document exit status.
Index: NEWS
===================================================================
RCS file: /home/sc/Repository/qm/NEWS,v
retrieving revision 1.25
diff -c -5 -p -r1.25 NEWS
*** NEWS 30 Jul 2003 22:25:00 -0000 1.25
--- NEWS 20 Aug 2003 18:36:17 -0000
*************** There is now a "-O" or "--outcomes" opti
*** 28,37 ****
--- 28,41 ----
outcomes, even when working within in the GUI.
There is a new "--result-stream" option to "qmtest run" and "qmtest
summarize" that allows users to specify custom result streams.
+ The exit code returned by QMTest is now documented, and now
+ distinguishes between a failure of QMTest to carry out the action
+ requested by the user and the failure of a test run by QMTest.
+
A defect that could cause a multi-threaded or multi-process tests run
to hang if a resource could not be set up has been corrected.
Comparisions between the expected standard output and actual standard
output done by test classes in the "command" module now ignore line
Index: qm/test/cmdline.py
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/cmdline.py,v
retrieving revision 1.92
diff -c -5 -p -r1.92 cmdline.py
*** qm/test/cmdline.py 10 Aug 2003 20:57:14 -0000 1.92
--- qm/test/cmdline.py 20 Aug 2003 18:36:18 -0000
*************** Valid formats are "full", "brief" (the d
*** 648,659 ****
error_occurred = 0
# Dispatch to the appropriate method.
if self.__command == "create-tdb":
! self.__ExecuteCreateTdb(db_path)
! return 0
method = {
"create" : self.__ExecuteCreate,
"create-target" : self.__ExecuteCreateTarget,
"extensions" : self.__ExecuteExtensions,
--- 648,658 ----
error_occurred = 0
# Dispatch to the appropriate method.
if self.__command == "create-tdb":
! return self.__ExecuteCreateTdb(db_path)
method = {
"create" : self.__ExecuteCreate,
"create-target" : self.__ExecuteCreateTarget,
"extensions" : self.__ExecuteExtensions,
*************** Valid formats are "full", "brief" (the d
*** 903,913 ****
else:
file = sys.stdout
# Write out the file.
qm.extension.write_extension_file(extension_class, arguments, file)
!
def __ExecuteCreateTdb(self, db_path):
"""Handle the command for creating a new test database.
'db_path' -- The path at which to create the new test database."""
--- 902,914 ----
else:
file = sys.stdout
# Write out the file.
qm.extension.write_extension_file(extension_class, arguments, file)
!
! return 0
!
def __ExecuteCreateTdb(self, db_path):
"""Handle the command for creating a new test database.
'db_path' -- The path at which to create the new test database."""
*************** Valid formats are "full", "brief" (the d
*** 937,947 ****
--- 938,950 ----
# Now process this just like "qmtest create".
self.__ExecuteCreate()
# Print a helpful message.
self._stdout.write(qm.message("new db message", path=db_path) + "\n")
+ return 0
+
def __ExecuteCreateTarget(self):
"""Create a new target file."""
# Make sure that the arguments are correct.
if (len(self.__arguments) < 2 or len(self.__arguments) > 3):
*************** Valid formats are "full", "brief" (the d
*** 1060,1069 ****
--- 1063,1074 ----
description += ("No description available: "
"could not load class.")
description += "\n\n"
self._stdout.write(qm.structured_text.to_text(description))
+
+ return 0
def __ExecuteRegister(self):
"""Register a new extension class."""
*************** Valid formats are "full", "brief" (the d
*** 1231,1241 ****
!= outcomes.get(r.GetId(), Result.PASS)):
any_unexpected_outcomes = 1
for s in streams:
s.Summarize()
! return any_unexpected_outcomes
def __ExecuteRemote(self):
"""Execute the 'remote' command."""
--- 1236,1249 ----
!= outcomes.get(r.GetId(), Result.PASS)):
any_unexpected_outcomes = 1
for s in streams:
s.Summarize()
! if any_unexpected_outcomes:
! return 2
!
! return 0
def __ExecuteRemote(self):
"""Execute the 'remote' command."""
*************** Valid formats are "full", "brief" (the d
*** 1364,1374 ****
# Run the tests.
engine = ExecutionEngine(database, test_ids, context, targets,
result_streams,
self.__GetExpectedOutcomes())
! return engine.Run()
def __ExecuteServer(self):
"""Process the server command."""
--- 1372,1385 ----
# Run the tests.
engine = ExecutionEngine(database, test_ids, context, targets,
result_streams,
self.__GetExpectedOutcomes())
! if engine.Run():
! return 2
!
! return 0
def __ExecuteServer(self):
"""Process the server command."""
Index: qm/test/qmtest.py
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/qmtest.py,v
retrieving revision 1.21
diff -c -5 -p -r1.21 qmtest.py
*** qm/test/qmtest.py 11 Aug 2003 06:43:16 -0000 1.21
--- qm/test/qmtest.py 20 Aug 2003 18:36:18 -0000
*************** def main():
*** 91,100 ****
--- 91,103 ----
########################################################################
# script
########################################################################
+ # Assume that something will go wrong.
+ exit_code = 1
+
try:
# Set the program name.
qm.common.program_name = "QMTest"
# Load messages.
*************** try:
*** 107,137 ****
exit_code = main()
except qm.cmdline.CommandError, msg:
print_error_message(msg)
sys.stderr.write(
"Run 'qmtest --help' to get instructions about how to use QMTest.\n")
- exit_code = 2
except qm.common.QMException, msg:
print_error_message(msg)
- exit_code = 1
except NotImplementedError:
exc_info = sys.exc_info()
method_name = traceback.extract_tb(exc_info[2])[-1][2]
print_error_message(qm.message("not implemented",
method_name = method_name))
sys.stderr.write(qm.common.format_traceback(exc_info))
- exit_code = 1
except KeyboardInterrupt:
- # User killed it; that's OK.
sys.stderr.write("\nqmtest: Interrupted.\n")
- exit_code = 0
except qm.platform.SignalException, se:
! # SIGTERM indicates a request to shut down.
! if se.GetSignalNumber() == signal.SIGTERM:
! exit_code = 1
! # Other signals should be handled earlier.
! else:
raise
finally:
# Collect garbage so that any "__del__" methods with externally
# visible side-effects are executed.
del qm.test.cmdline._the_qmtest
--- 110,133 ----
exit_code = main()
except qm.cmdline.CommandError, msg:
print_error_message(msg)
sys.stderr.write(
"Run 'qmtest --help' to get instructions about how to use QMTest.\n")
except qm.common.QMException, msg:
print_error_message(msg)
except NotImplementedError:
exc_info = sys.exc_info()
method_name = traceback.extract_tb(exc_info[2])[-1][2]
print_error_message(qm.message("not implemented",
method_name = method_name))
sys.stderr.write(qm.common.format_traceback(exc_info))
except KeyboardInterrupt:
sys.stderr.write("\nqmtest: Interrupted.\n")
except qm.platform.SignalException, se:
! # SIGTERM indicates a request to shut down. Other signals
! # should be handled earlier.
! if se.GetSignalNumber() != signal.SIGTERM:
raise
finally:
# Collect garbage so that any "__del__" methods with externally
# visible side-effects are executed.
del qm.test.cmdline._the_qmtest
Index: qm/test/classes/mount_database.py
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/classes/mount_database.py,v
retrieving revision 1.2
diff -c -5 -p -r1.2 mount_database.py
*** qm/test/classes/mount_database.py 13 Apr 2003 05:57:29 -0000 1.2
--- qm/test/classes/mount_database.py 20 Aug 2003 18:36:18 -0000
*************** class MountDatabase(Database):
*** 33,43 ****
database with a mount point of "y" has the ID "x.y" in the
containing database.
The contained databases are found by looking for subdirectories of
the 'MountDatabase' directory. Every immediate subdirectory which
! is itself a QMTest database is mounted; it's mount point is the
name of the subdirectory."""
class MountedSuite(Suite):
"""A 'MountedSuite' is a suite from a mounted database."""
--- 33,43 ----
database with a mount point of "y" has the ID "x.y" in the
containing database.
The contained databases are found by looking for subdirectories of
the 'MountDatabase' directory. Every immediate subdirectory which
! is itself a QMTest database is mounted; its mount point is the
name of the subdirectory."""
class MountedSuite(Suite):
"""A 'MountedSuite' is a suite from a mounted database."""
Index: qm/test/doc/reference.xml
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/doc/reference.xml,v
retrieving revision 1.29
diff -c -5 -p -r1.29 reference.xml
*** qm/test/doc/reference.xml 15 Aug 2003 09:05:23 -0000 1.29
--- qm/test/doc/reference.xml 20 Aug 2003 18:36:19 -0000
***************
*** 1557,1566 ****
--- 1557,1588 ----
</glossdef>
</glossentry>
</glosslist>
</section> <!-- sec-config-file -->
+
+ <section id="sec-return-value">
+ <title>Return Value</title>
+
+ <para>If QMTest successfully performed the action requested,
+ &qmtest; returns 0. For the <command>&qmtest-cmd; run</command> or
+ <command>&qmtest-cmd; summarize</command> commands, success implies
+ not only that the tests ran, but also that all of the tests passed
+ (if the <option>&dashdash;outcomes</option> option was not used) or
+ had their expected outcomes (if the
+ <option>&dashdash;outcomes</option> option was used).</para>
+
+ <para>If QMTest could not perform the action requested,
+ &qmtest-cmd; returns 1.</para>
+
+ <para>If either the <command>run</command> command or the
+ <command>summarize</command> command was used, and at least one
+ test failed (if the <option>&dashdash;outcomes</option> option was
+ not used) or had an unexpected outcome (if the
+ <option>&dashdash;outcomes</option> option was used), &qmtest-cmd;
+ returns 2.</para>
+ </section> <!-- sec-return-value -->
</section>
<section id="sec-ref-classes">
<title>Test and Resource Classes</title>
More information about the qmtest
mailing list