should versions be unicode?

[warner]

So I'm working on the "style" branch, to make it possible to configure the style of version string that you want, and I've run into a question that I need help with. Should the version string exposed to your program, on python2, be bytes or unicode? How about on python3?

In earlier versions, I think we've been doing bytes on py2 and str (ie unicode) on py3. In a sense, these are the "normal" printable-string types on those platforms. Versions are frequently printed in response to a CLI command, sometimes stored in files (debug logs, sometimes as a compatibility marker for serialized data), and occasionally sent over the network (again for compatibility).

Two things are affecting my viewpoint:

• the new branch uses JSON to store version information for the "from-file" case (where you've run setup.py build and it writes pre-rendered version data into the _version.py), and python's modern JSON module likes to return unicode keys and values on py2, even if you wrote bytes in the first place.
• git tags can nominally be unicode, directory-name prefixes could nominally be unicode, and we derive versions from both of those. So maybe versions should be unicode too.

The switch might require applications to explicitly .encode() the version before sending it to something that really wants bytes, like a logfile or (maybe even) stdout.

Thoughts?

[warner]

Oh, also, should the full git hex-revisionid be unicode too? It's obviously always going to fit in ASCII, but for consistency? The new "style" branch provides a get_versions() function that returns a dictionary with exactly four keys:

• version: the usual rendered version string
• full-revisionid: the hex SHA1 hash (or None)
• dirty: a boolean
• error: a string, or None, to signal an error

In my current branch, the dictionary keys are unicode, version is always unicode, full-revisionid is either unicode or None, and error is either unicode or None. Having unicode keys isn't a big deal, since things seem to compare equal (no need to write ungainly x=get_versions()[u"version"] in py2), but still feels a bit funky.

[zooko]

You asked for opinions on this, so here's mine:

As you already know, I think the following two ideas are wrong wrong wrong and should be stamped out:

• "strings (i.e. bytes) are the normal type to hold printable things on Py2" — no no no, that was a terrible mistake which has cost everyone a great amount of time and labor, and we should struggle to stamp out this troublesome legacy practice. unicode objects are the only correct type to hold printable things in Py2.

By the way, this mistake and the wrongly-executed attempt to fix it are the reason that Py3 is backwards-incompatible with Py2. Let's not waste the Python community's terrible sacrifice by promulgating the old, wrong way of doing things. ;-)

• "git hexids are always going to fit in ascii, so they might as well be held in bytes" — no, no, no! That's the wrong type. That's like saying "These pointers are always going to fit into unsigned longs, so I might as well store them in unsigned longs." Even if that sentence is correct, doing it that way is incorrectly typed, confusing to other programmers, and error-prone.

So my answer is that your current "style" branch is correct: everything that holds a natural-language or printable string is unicode (or None).

Oh, oops, why is error a string? That should be a unicode.

[effigies]

Obviated by dropping Python 2.