Patches for scheduler updates
Nathaniel Smith
njs at pobox.com
Thu Aug 7 09:36:54 UTC 2003
Attached is a patch that fixes a few bugs in the new scheduler, and in
the test suite that failed to catch said bugs.
The changes to the scheduler are pretty minor -- a few obvious bugs,
and a change to the idle target handling to make it a bit less clever.
The changes to the test suite are so that when we, e.g., run the
check-threads test, the regression tests will also be run using the
thread target. Previously, though we ran the regression tests
themselves in parallel, we only ever tested qmtest run in serial...
this fixes that.
However, this has odd effects. If the results are to be believed,
then when running with threaded parallelism the target checking
doesn't work at all (regress.bad_target1 and regress.bad_target2
fail -- apparently we don't notice that the tests aren't supposed to
be run and run them anyway?), and when running with process parallelism
then lots and lots of things don't work. Like, most but not all of
the regression tests fail, and it's not immediately obvious to me why.
I also get exactly the same results from my original scheduler, the
version in HEAD after applying the fixes here, and even a checkout
from last month before the scheduler changes. Which is weird, since
the bad target checking in particular is totally different in all
three cases...
Tomorrow I'll make an attempt at digging into these failures in the
hopes of figuring out what's going on, but thought people might like a
heads up and a chance to try to reproduce this... or point out the
pilot error causing all this ;-).
-- Nathaniel
--
"But suppose I am not willing to claim that. For in fact pianos
are heavy, and very few persons can carry a piano all by themselves."
-------------- next part --------------
diff -urN --exclude='*~' --exclude='.*' --exclude=CVS --exclude='*.pyo' --exclude='*.pyc' --exclude=build --exclude=GNUmakefile --exclude=config.log --exclude=config.status --exclude=setup_path.py --exclude=qm.sh --exclude=qmtest --exclude=qm.spec --exclude='*.dtd' --exclude=CATALOG --exclude=thread_target --exclude=process_target --exclude='*.qmr' qm-clean/ChangeLog qm-efficient-scheduling/ChangeLog
--- qm-clean/ChangeLog 2003-08-06 13:00:29.000000000 -0700
+++ qm-efficient-scheduling/ChangeLog 2003-08-07 01:58:52.000000000 -0700
@@ -1,3 +1,23 @@
+2003-08-06 Nathaniel Smith <njs at codesourcery.com>
+
+ * qm/test/execution_engine.py: Some comment updates. Also:
+ (ExecutionEngine._TARGET_IDLE): New variable.
+ (ExecutionEngine._TARGET_BUSY): Likewise.
+ (ExecutionEngine._TARGET_STARVING): Likewise.
+ (ExecutionEngine.__num_idle_targets): Replace with...
+ (ExecutionEngine.__has_idle_targets): ...this.
+ (ExecutionEngine._RunTests): Incorporate above changes.
+ (ExecutionEngine.__AddResult): Likewise.
+
+ (ExecutionEngine.__AddToTargetPatternQueue): Fix typo.
+
+ * GNUMakefile.in (check-threads): Set new context variable
+ qmtest_target.
+ (check-processes): Likewise.
+ (check-rsh): Likewise.
+ * tests/regress/QMTest/selftest.py (RegTest.Run): Use context
+ variable qmtest_target if defined.
+
2003-08-01 Mark Mitchell <mark at codesourcery.com>
* qm/test/base.py (split_results_by_expected_outcome): Remove.
diff -urN --exclude='*~' --exclude='.*' --exclude=CVS --exclude='*.pyo' --exclude='*.pyc' --exclude=build --exclude=GNUmakefile --exclude=config.log --exclude=config.status --exclude=setup_path.py --exclude=qm.sh --exclude=qmtest --exclude=qm.spec --exclude='*.dtd' --exclude=CATALOG --exclude=thread_target --exclude=process_target --exclude='*.qmr' qm-clean/GNUmakefile.in qm-efficient-scheduling/GNUmakefile.in
--- qm-clean/GNUmakefile.in 2003-08-06 13:00:29.000000000 -0700
+++ qm-efficient-scheduling/GNUmakefile.in 2003-08-07 01:30:22.000000000 -0700
@@ -227,7 +227,8 @@
thread thread_target.ThreadTarget
qm/test/qmtest -D tests run -T tests/QMTest/thread_target \
$(QMTESTFLAGS) \
- -c qmtest_path=qm/test/qmtest
+ -c qmtest_path=qm/test/qmtest \
+ -c qmtest_target=tests/QMTest/thread_target
check-processes: all
rm -f tests/QMTest/process_target
@@ -236,7 +237,8 @@
process process_target.ProcessTarget
qm/test/qmtest -D tests run -T tests/QMTest/process_target \
$(QMTESTFLAGS) \
- -c qmtest_path=qm/test/qmtest
+ -c qmtest_path=qm/test/qmtest \
+ -c qmtest_target=tests/QMTest/process_target
check-rsh: all
rm -f tests/QMTest/rsh_target
@@ -246,7 +248,8 @@
rsh rsh_target.RSHTarget
qm/test/qmtest -D tests run -T tests/QMTest/rsh_target \
$(QMTESTFLAGS) \
- -c qmtest_path=`pwd`/qm/test/qmtest
+ -c qmtest_path=`pwd`/qm/test/qmtest \
+ -c qmtest_target=`pwd`/tests/QMTest/rsh_target
########################################################################
# Documentation Rules
diff -urN --exclude='*~' --exclude='.*' --exclude=CVS --exclude='*.pyo' --exclude='*.pyc' --exclude=build --exclude=GNUmakefile --exclude=config.log --exclude=config.status --exclude=setup_path.py --exclude=qm.sh --exclude=qmtest --exclude=qm.spec --exclude='*.dtd' --exclude=CATALOG --exclude=thread_target --exclude=process_target --exclude='*.qmr' qm-clean/qm/test/execution_engine.py qm-efficient-scheduling/qm/test/execution_engine.py
--- qm-clean/qm/test/execution_engine.py 2003-08-06 17:45:21.000000000 -0700
+++ qm-efficient-scheduling/qm/test/execution_engine.py 2003-08-07 01:22:14.000000000 -0700
@@ -164,6 +164,11 @@
+ _TARGET_IDLE = "IDLE"
+ _TARGET_BUSY = "BUSY"
+ _TARGET_STARVING = "STARVING"
+
+
def __init__(self,
database,
test_ids,
@@ -315,14 +320,12 @@
# Every target is in one of three states: busy, idle, or
# starving. A busy target is running tests, an idle target is
# ready to run tests, and a starving target is ready to run
- # tests, but no tests are available for it to run. The value
- # recorded in the table is 'None' for a starving target, true
- # for an idle target, and false for a busy target.
+ # tests, but no tests are available for it to run.
self.__target_state = {}
+ # All targets are initially idle.
for target in self.__targets:
- self.__target_state[target] = 1
- # The total number of idle targets.
- self.__num_idle_targets = len(self.__targets)
+ self.__target_state[target] = self._TARGET_IDLE
+ self.__has_idle_targets = 1
# Figure out what target groups are available.
self.__target_groups = {}
@@ -345,7 +348,7 @@
pass
# Now look for idle targets.
- if self.__num_idle_targets == 0:
+ if not self.__has_idle_targets:
# Block until one of the running tests completes.
self._Trace("All targets are busy -- waiting.")
self.__CheckForResponse(wait=1)
@@ -354,19 +357,22 @@
# Go through each of the idle targets, finding work for it
# to do.
- self.__num_idle_targets = 0
+ self.__has_idle_targets = 0
for target in self.__targets:
- if self.__target_state[target] != 1:
+ if self.__target_state[target] != self._TARGET_IDLE:
continue
# Try to find work for the target. If there is no
# available work, the target is starving.
if not self.__FeedTarget(target):
- self.__target_state[target] = None
+ self.__target_state[target] = self._TARGET_STARVING
else:
- is_idle = target.IsIdle()
- self.__target_state[target] = is_idle
- if is_idle:
- self.__num_idle_targets += 1
+ # We gave the target some work, which may have
+ # changed its idle state, so update the status.
+ if target.IsIdle():
+ self.__target_state[target] = self._TARGET_IDLE
+ self.__has_idle_targets = 1
+ else:
+ self.__target_state[target] = self._TARGET_BUSY
# Now all tests have been started; we just have wait for them
# all to finish.
@@ -384,20 +390,19 @@
self._Trace("Looking for a test for target %s" % target.GetName())
- descriptor = None
-
# See if there is already a ready-to-run test for this target.
for pattern in self.__patterns.get(target.GetGroup(), []):
tests = self.__target_pattern_queues.get(pattern, [])
if tests:
descriptor = tests.pop()
break
-
- # If there is no ready test, find one.
- descriptor = self.__FindRunnableTest(target)
- if descriptor is None:
- # There are no more tests ready to run.
- return 0
+ else:
+ # There was no ready-to-run test queued, so try to find one
+ # another one.
+ descriptor = self.__FindRunnableTest(target)
+ if descriptor is None:
+ # There really are no more tests ready to run.
+ return 0
target_name = target.GetName()
test_id = descriptor.GetId()
@@ -451,8 +456,8 @@
# A->B, A->C, B->C, where we can't know ahead of time
# that A's dependence on C is unnecessary.
if self.__statuses[new_test_id].HasBeenQueued():
- # This one is already in process. This is also what
- # a dependency cycle looks like, so check for that
+ # This one is already in process. This might
+ # indicate a dependency cycle, so check for that
# now.
if new_test_id in self.__ids_on_stack:
self._Trace("Cycle detected (%s)"
@@ -552,7 +557,7 @@
self.__pattern_ok[pattern] = 0
for group in self.__target_groups:
if re.match(pattern, group):
- self.__pattern_ok[group] = 1
+ self.__pattern_ok[pattern] = 1
patterns = self.__patterns.setdefault(group, [])
patterns.append(pattern)
# If none of the targets can run this test, mark it untested.
@@ -581,7 +586,8 @@
try:
prereq_status = self.__statuses[prereq_id]
except KeyError:
- # This prerequisite is not being run at all.
+ # This prerequisite is not being run at all, so skip
+ # it.
continue
if prereq_status.IsFinished():
@@ -634,8 +640,8 @@
if (target and target.IsIdle()):
# Output a trace message.
self._Trace("Target is now idle.\n")
- self.__target_state[target] = 1
- self.__num_idle_targets += 1
+ self.__target_state[target] = self._TARGET_IDLE
+ self.__has_idle_targets = 1
# Only tests have expectations or scheduling dependencies.
if result.GetKind() == Result.TEST:
@@ -670,8 +676,8 @@
# Any targets that were starving may now be able to find
# work.
for t in self.__targets:
- if self.__target_state[t] is None:
- self.__target_state[t] = 1
+ if self.__target_state[t] == self._TARGET_STARVING:
+ self.__target_state[t] = self._TARGET_IDLE
# Output a trace message.
self._Trace("Writing result for %s to streams." % id)
diff -urN --exclude='*~' --exclude='.*' --exclude=CVS --exclude='*.pyo' --exclude='*.pyc' --exclude=build --exclude=GNUmakefile --exclude=config.log --exclude=config.status --exclude=setup_path.py --exclude=qm.sh --exclude=qmtest --exclude=qm.spec --exclude='*.dtd' --exclude=CATALOG --exclude=thread_target --exclude=process_target --exclude='*.qmr' qm-clean/tests/regress/QMTest/selftest.py qm-efficient-scheduling/tests/regress/QMTest/selftest.py
--- qm-clean/tests/regress/QMTest/selftest.py 2003-03-23 23:24:24.000000000 -0800
+++ qm-efficient-scheduling/tests/regress/QMTest/selftest.py 2003-08-07 02:13:27.000000000 -0700
@@ -68,7 +68,12 @@
# The QMTest binary to test is specified as a context variable.
qmtest = context['qmtest_path']
+ # Set the basic argument vector.
argv = (qmtest, "-D", path, "run", "-O", results, "-o", output)
+
+ # If the context also specifies a target, add that.
+ if context.has_key("qmtest_target"):
+ argv += ("-T", context["qmtest_target"])
e = qm.executable.RedirectedExecutable()
status = e.Run(argv)
More information about the qmtest
mailing list