[qmtest] Unicode support

[Pavel Cisar]

Hi,

The issue with unicode annotations was my fault, sorry. After the fix, I 
can create and run tests with unicode without any change to QMTest code, 
although special support in extension classes is needed.

On 25 Oct 2004 at 14:55, Mark Mitchell wrote:

> We're definitely interested in supporting Unicode.  If you have concrete 
> examples of places where we fail to do that, that would help.  For 
> example, if we can recreate the problem with a simple test using a 
> built-in QMTest test class, that will help us to solve it.

Next notes apply to latest QMTest stable build.

1. QMtest can flawlessly store and load tests that have unicode in 
attributes (thanks to unicode support in XML handling).

2. There are two main points of failure:

a) Inadequate charset specification in qm.web.DtmlPage.GenerateHtmlHeader
It's iso-8859-1 instead utf-8. Next code in our extension module amend 
it:

_ORIGINAL_IMPL = qm.web.DtmlPage.GenerateHtmlHeader

def GenerateHtmlHeader(self, description, headers=''):
    header = _ORIGINAL_IMPL(self, description, headers=headers)
    header = header.replace('charset=iso-8859-1', 'charset=utf-8')
    return header

qm.web.DtmlPage.GenerateHtmlHeader = GenerateHtmlHeader

b) qm.fields.TextField doesn't support unicode at all (it fails on use of 
str() function). Next customized UnicodeField fixes that.

class UnicodeField(qm.fields.TextField):
    """
      A subclass of 'TextField' that supports Unicode.
    """

    def FormatValueAsHtml(self, server, value, style, name=None):
        # $value is typically unicode rather than str.  In any case, we 
always
        # encode the returned HTML code in UTF-8.
        #
        # qm.fields.TextField.FormatValueAsHtml can't handle Unicode, so
        # we replace $value with a flag, call the super-method, convert
        # the resulting HTML str into a unicode object, replace the flag
        # with the real $value, then encode $value in UTF-8 for
        # transmission to the client.
        flag = '_______REPLACE_THIS_FLAG_______'
        html = qm.fields.TextField.FormatValueAsHtml(self, server, flag, 
style,
            name=name
          )
        # Next try-except is necessary for QMTest. When test is loaded 
for edit,
        # the value content is ok, but if test editing form contains 
button
        # (for example to add/remove substitution definition) and this 
button is
        # used, the form reload logic goes in different route, and value 
has encoding
        # that replace cannot handle. This case is handled by except 
part.
        try:
          html = unicode(html,'UTF8').replace(flag, value)
        except UnicodeError:
          html = unicode(html,'UTF8').replace(flag, 
unicode(value,'UTF8'))
            
        return html.encode('UTF8')

This is only basic text field. Our test class for complex tests that are 
written in Python is far more complex. Anyone interested can find it 
(soon) in our CVS at sourceforge.

3. It's necessary to keep all unicode values that are displayed (like 
annotations) encoded as UTF8.

> (Also, in the usual open-source way, we'll be likely to do things 
> quickest for our paying customers; if you'd like to hire us to work on 
> this project, we'd be happy to talk to you about that.)

Huh, because we are an open source project, it's somehow complicated for 
us to become a paying customer :-)

Best regards
Pavel Cisar (ICQ: 89017288)
http://www.ibphoenix.com
For all your upto date Firebird and
InterBase information