[qmtest] RFE: distinguish qmtest failure and test failures in qmtest's return value

Wed Aug 20 19:47:46 UTC 2003

On Wed, 2003-08-20 at 12:15, Zack Weinberg wrote:
> Mark Mitchell <mark at codesourcery.com> writes:
> 
> > I checked in the attached patch, which makes QMTest return 2 for failing
> > tests and 1 for all other errors.  I've also documented this behavior;
> > until now the documentation made no promises about the return code.
> 
> Can I suggest it be the other way around - 1 for failing tests, 2 for
> errors?  That is consistent with, for instance, GNU cmp.

Sure; patch committed.

-- 
Mark Mitchell
CodeSourcery, LLC
mark at codesourcery.com
-------------- next part --------------
2003-08-20  Mark Mitchell  <mark at codesourcery.com>

	* qm/test/cmdline.py (QMTest.__ExecuteCreate): Return 2 for
	failure, not 1.
	(QMTest.__ExecuteCreateTdb): Likewise.
	(QMTest.__ExecuteCreateTarget): Likewise.
	(QMTest.__ExecuteExtensions): Likewise.
	(QMTest.__ExecuteRegister): Likewise.
	(QMTest.__ExecuteSummarize): Return 2 when a test fails, not 1.
	(QMTest.__ExecuteRun): Likewise.
	* qm/test/qmtest.py: Use 2 as the default exit code, not 1.
	* qm/test/doc/reference.xml: Update accordingly.

Index: qm/test/cmdline.py
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/cmdline.py,v
retrieving revision 1.94
diff -c -5 -p -r1.94 cmdline.py
*** qm/test/cmdline.py	20 Aug 2003 18:46:57 -0000	1.94
--- qm/test/cmdline.py	20 Aug 2003 19:37:56 -0000
*************** Valid formats are "full", "brief" (the d
*** 852,862 ****
          """Create a new extension file."""

          # Check that the right number of arguments are present.
          if len(self.__arguments) != 2:
              self.__WriteCommandHelp("create")
!             return 1

          # Figure out what database (if any) we are using.
          try:
              database = self.GetDatabase()
          except:
--- 852,862 ----
          """Create a new extension file."""

          # Check that the right number of arguments are present.
          if len(self.__arguments) != 2:
              self.__WriteCommandHelp("create")
!             return 2

          # Figure out what database (if any) we are using.
          try:
              database = self.GetDatabase()
          except:
*************** Valid formats are "full", "brief" (the d
*** 901,911 ****

          'db_path' -- The path at which to create the new test database."""

          if len(self.__arguments) != 0:
              self.__WriteCommandHelp("create-tdb")
!             return 1

          # Create the directory if it does not already exists.
          if not os.path.isdir(db_path):
              os.mkdir(db_path)
          # Create the configuration directory.
--- 901,911 ----

          'db_path' -- The path at which to create the new test database."""

          if len(self.__arguments) != 0:
              self.__WriteCommandHelp("create-tdb")
!             return 2

          # Create the directory if it does not already exists.
          if not os.path.isdir(db_path):
              os.mkdir(db_path)
          # Create the configuration directory.
*************** Valid formats are "full", "brief" (the d
*** 935,945 ****
          """Create a new target file."""

          # Make sure that the arguments are correct.
          if (len(self.__arguments) < 2 or len(self.__arguments) > 3):
              self.__WriteCommandHelp("create-target")
!             return 1

          # Pull the required arguments out of the command line.
          target_name = self.__arguments[0]
          class_name = self.__arguments[1]
          if (len(self.__arguments) > 2):
--- 935,945 ----
          """Create a new target file."""

          # Make sure that the arguments are correct.
          if (len(self.__arguments) < 2 or len(self.__arguments) > 3):
              self.__WriteCommandHelp("create-target")
!             return 2

          # Pull the required arguments out of the command line.
          target_name = self.__arguments[0]
          class_name = self.__arguments[1]
          if (len(self.__arguments) > 2):
*************** Valid formats are "full", "brief" (the d
*** 1012,1022 ****
          """List the available extension classes."""

          # Check that the right number of arguments are present.
          if len(self.__arguments) != 0:
              self.__WriteCommandHelp("extensions")
!             return 1

          try:
              database = self.GetDatabase()
          except:
              # If the database could not be opened that's OK; this
--- 1012,1022 ----
          """List the available extension classes."""

          # Check that the right number of arguments are present.
          if len(self.__arguments) != 0:
              self.__WriteCommandHelp("extensions")
!             return 2

          try:
              database = self.GetDatabase()
          except:
              # If the database could not be opened that's OK; this
*************** Valid formats are "full", "brief" (the d
*** 1061,1071 ****
          """Register a new extension class."""

          # Make sure that the KIND and CLASS were specified.
          if (len(self.__arguments) != 2):
              self.__WriteCommandHelp("register")
!             return 1
          kind = self.__arguments[0]
          class_name = self.__arguments[1]

          # Check that the KIND is valid.
          if kind not in base.extension_kinds:
--- 1061,1071 ----
          """Register a new extension class."""

          # Make sure that the KIND and CLASS were specified.
          if (len(self.__arguments) != 2):
              self.__WriteCommandHelp("register")
!             return 2
          kind = self.__arguments[0]
          class_name = self.__arguments[1]

          # Check that the KIND is valid.
          if kind not in base.extension_kinds:
*************** Valid formats are "full", "brief" (the d
*** 1225,1235 ****
                      any_unexpected_outcomes = 1
          for s in streams:
              s.Summarize()

          if any_unexpected_outcomes:
!             return 2

          return 0

      def __ExecuteRemote(self):
--- 1225,1235 ----
                      any_unexpected_outcomes = 1
          for s in streams:
              s.Summarize()

          if any_unexpected_outcomes:
!             return 1

          return 0

      def __ExecuteRemote(self):
*************** Valid formats are "full", "brief" (the d
*** 1361,1371 ****
          # Run the tests.
          engine = ExecutionEngine(database, test_ids, context, targets,
                                   result_streams,
                                   self.__GetExpectedOutcomes())
          if engine.Run():
!             return 2

          return 0

      def __ExecuteServer(self):
--- 1361,1371 ----
          # Run the tests.
          engine = ExecutionEngine(database, test_ids, context, targets,
                                   result_streams,
                                   self.__GetExpectedOutcomes())
          if engine.Run():
!             return 1

          return 0

      def __ExecuteServer(self):
Index: qm/test/qmtest.py
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/qmtest.py,v
retrieving revision 1.22
diff -c -5 -p -r1.22 qmtest.py
*** qm/test/qmtest.py	20 Aug 2003 18:46:57 -0000	1.22
--- qm/test/qmtest.py	20 Aug 2003 19:37:57 -0000
*************** def main():
*** 92,102 ****
  ########################################################################
  # script
  ########################################################################

  # Assume that something will go wrong.
! exit_code = 1

  try:
      # Set the program name.
      qm.common.program_name = "QMTest"

--- 92,102 ----
  ########################################################################
  # script
  ########################################################################

  # Assume that something will go wrong.
! exit_code = 2

  try:
      # Set the program name.
      qm.common.program_name = "QMTest"

Index: qm/test/doc/reference.xml
===================================================================
RCS file: /home/sc/Repository/qm/qm/test/doc/reference.xml,v
retrieving revision 1.31
diff -c -5 -p -r1.31 reference.xml
*** qm/test/doc/reference.xml	20 Aug 2003 18:46:57 -0000	1.31
--- qm/test/doc/reference.xml	20 Aug 2003 19:37:58 -0000
***************
*** 1558,1576 ****
     not only that the tests ran, but also that all of the tests passed
     (if the <option>&dashdash;outcomes</option> option was not used) or
     had their expected outcomes (if the
     <option>&dashdash;outcomes</option> option was used).</para>

-    <para>If QMTest could not perform the action requested,
-    &qmtest-cmd; returns 1.</para>
- 
     <para>If either the <command>run</command> command or the
     <command>summarize</command> command was used, and at least one
     test failed (if the <option>&dashdash;outcomes</option> option was
     not used) or had an unexpected outcome (if the
     <option>&dashdash;outcomes</option> option was used), &qmtest-cmd;
!    returns 2.</para> 
    </section> <!-- sec-return-value -->

   </section>

   <section id="sec-ref-classes">
--- 1558,1577 ----
     not only that the tests ran, but also that all of the tests passed
     (if the <option>&dashdash;outcomes</option> option was not used) or
     had their expected outcomes (if the
     <option>&dashdash;outcomes</option> option was used).</para>

     <para>If either the <command>run</command> command or the
     <command>summarize</command> command was used, and at least one
     test failed (if the <option>&dashdash;outcomes</option> option was
     not used) or had an unexpected outcome (if the
     <option>&dashdash;outcomes</option> option was used), &qmtest-cmd;
!    returns 1.</para>
! 
!    <para>If QMTest could not perform the action requested,
!    &qmtest-cmd; returns 2.</para>
! 
    </section> <!-- sec-return-value -->

   </section>

   <section id="sec-ref-classes">