[qmtest] [PATCH] Removing ExtensionClass dependency

[Nathaniel Smith]

I'll try to respond to several of the issues brought up in this thread
at once.

On Sun, Apr 13, 2003 at 01:43:43PM -0700, Mark Mitchell wrote:
> There are a bunch of things we need to do to make QMTest a "proper"
> Python module, including (at least):
> 
> (1) Deal with the setup_path.py stuff that adds various things to the
> Python search path.  Instead, package things up so that Python can just
> find them.
> 
> (2) Avoid stomping on existing Python stuff when QMTest is installed.
> 
> Nathaniel's patch works on both of these at once, I think:

Yes.

> - The bits that change the import statements in DTML where Nathaniel has
> "subltety" comments are (I think) trying to work on (1).  (Because
> Nathaniel moved things to "external", because that directory didn't get
> added to the Python path by setup_path.py.)  But maybe I misunderstand?
>
> - Moving things to qm.external so that installing QMTest in the "usual
> place", where DTML is already installed, works out OK.

These complement each other.  The overall idea is that after applying
my patch, it will become possible to install qm to one of the standard
python package directories, instead of /usr/local/lib/qm.  In
particular, we won't have to modify the Python search path when
starting up qmtest.  We don't _have_ to rearrange things to make this
possible; we could have just changed the build system to put
everything in site-packages/ and called it done.  If we did that,
though, then DocumentTemplate and ExtensionClass would have caused
problems, because the build system doesn't check to see if they're
already installed, and ships with weird old versions that you wouldn't
want to use generally anyway.  The solution was to effectively rename
the DocumentTemplate package to qm.external.DocumentTemplate.  So
that's what my patch does.  (Note that this is different from simply
storing the DocumentTemplate code in qm/external/ and adding
qm/external/ to the Python path, since it lets us avoid tweaking the
Python path altogether.)

The "subtlety" bit is fixing an obscure bug little bug in
DocumentTemplate.DT_String that would otherwise prevent the rename
from working.  What that DT_String effectively does is
  try:  exec 'import DT_Foo' in d
  except ImportError: exec 'import DocumentTemplate.DT_Foo' in d
  handle = d["DT_Foo"]
as a way of getting a handle to the DT_Foo module.  This looks like it
should work, even after renaming DocumentTemplate, because python
should interpret the first import statement as being relative to
"whatever package contains this code", no matter whether that's called
DocumentTemplate or qm.external.DocumentTemplate.  What really
happens, though, is that the exec executes in some default context,
and the relative import always fails, and then the absolute import
fails because we've renamed DocumentTemplate.  This code is totally
wrong as stands; it looks like someone at Zope noticed that the
relative import was failing, threw in the try block as a hacky patch,
and no-one ever noticed.  Fortunately, there's an easy fix: by passing
the current module's globals to exec, Python's import mechanism is
able to deduce the current package, and the relative import works.
Hence my use of
  exec 'import DT_Foo' in globals(), d
After this change, no DocumentTemplate code relies on the name of its
containing package.

> I'd rather have a clear plan before we dive in here.  It seems
> reasonable to deal with (2) without dealing with (1) first, to me.

Yes, that's more-or-less what I'm trying to do -- to clear the ground
so that it becomes feasible to kill setup_path.py as well.  The
overall roadmap in my head is something like:
  1) Kill ExtensionClass, move DocumentTemplate -- at this point,
     qmtest becomes a single pure-Python module.
  2) Change the build system to install qmtest into the normal Python
     path, and remove the path rewriting stuff in setup_path.py.
  3) Rewrite the directory finding stuff (qm/common.py,
     qm/setup_path.py.in) to get rid of the dependency on environment
     variables.
  4) (Optional) Take this opportunity to clean up qmtest's startup,
     since qm.sh has become largely superfluous.  (See
     /usr/bin/happydoc for an example of how simple bootstrapping a
     Python program can be.)
This way, any given change is incremental (and conceivably doable in
the drabs of free time I have available :-)), but by such small steps
we end up with a standard, well-behaved Python app/library.

> I guess you're thinking about how to make a GNU/Linux binary package
> that works *whether or not* DTML is already installed.  I'm wondering if
> we need to deal with that complexity, or if we can just have two binary
> packages: one that contains DTML and one that expects you've already got
> it installed.

I'm not sure that this is the best idea.  I think in the ideal world,
qmtest itself wouldn't ship with any external code at all, and would
be documented to depend on DocumentTemplate, and the Debian package
would depend on python-documenttemplate, and so on, and everything
would be neat and clean.  (And then, of course, you could have your
qmtest-inna-box distribution, that contained Python, supporting
modules, and so on.)  

Like much Zope code, though, it's not actually convenient to install
DocumentTemplate on its own; AFAICT there are no .deb's or .rpm's
available, and the zope.org folks aren't generally worried about
making it easy to get just pieces of Zope -- apparently you must use
CVS, and then there are undocumented dependencies on other pieces of
Zope, and it's generally a headache.  Considering this, keeping a
local copy of DocumentTemplate in qm.external seems reasonable.  (I'm
not completely convinced of this, since there's a danger of it
becoming a maintainence burden (it still uses regex.py and regsub.py,
for instance, which were deprecated in python 1.5!), but this is a
somewhat separate issue, and can always be dealt with later, if
maintaining it does become a hassle.)

At that point, though, I'm not sure what the point of looking for
local installs of DocumentTemplate are; if qmtest is going to be
carrying around a known-good version anyway, why sometimes throw it
out in favor of some random package that may or may not work?  Any
pre-existing DocumentTemplate is _likely_ to work, of course, and I
suppose skipping the install of DocumentTemplate could save a few KiB
on systems that already have Zope installed (i.e., systems that
probably aren't hurting for disk space), but this seems a pretty small
gain to trade for an increase in complexity and possible support
headaches.

> I'm also a little bit confused because DTML is already installed in
> lib/qm/zope-dtml, so it's not actually going to clash with anything
> already installed as far as I can tell.

Well, as above, the goal is to change that :-).

Hopefully this clarifies things a bit...
-- Nathaniel

-- 
"...All of this suggests that if we wished to find a modern-day model
for British and American speech of the late eighteenth century, we could
probably do no better than Yosemite Sam."

This email may be read aloud.