Tim,
Thanks for this note. It gives me a better understanding of your
motivation, something that, IMO, should be more clear in the
document. Let me make some observations that may clarify the
difference in perspective for others and that might suggest some
adjustments in the document. You (and the community) may not go
where this note suggests, but it might be worth considering as a
perspective. It might also help with details about the container /
protocol slot distinction Peter asked about. Or not.
I assume you and Paul know everything in the next few paragraphs, but
for context and for others evaluating the document and the various
reviews...
I think it is clear, at least to anyone who has spent even a bit of
time studying the issues, that composing strings for others to use
(especially in comparison with things they already have or know)
presents huge opportunities for mishap, mischief, or worse. We
usually assume that identifiers are more susceptible to problems than
free-text strings but maybe that is just because perceiving and using
one identifier when another was intended is likely to have worse
consequences than "just" misreading a string. Homographs are the
most publicized of the problems, but, depending on the character set
encoding and choices of display type families ("fonts"), anyone who
has confused a digit "1" character with a lower-case "L" should
understand they can occur even if the repertoire is limited to
ASCII. Whether homographs are more important than other potential
problems is debatable, but the outcome of that debate depends on
circumstances.
PRECIS, IDNA2008 (for the rather specific IDN case), the ICANN MSR
and LGR work (for even more specific cases), and parts of the W3C and
Unicode specs are all focused on minimizing the risks of those
problems. Put more positively, those specs are about increasing the
odds that a string chosen by a naive person for use as an identifier
will perform well and that text strings transmitted by one human to
another will be seen, heard, and understood as intended. As with
the trivial "1"/"l" problem (or opportunity from malicious behavior),
no collection of rules or list of code points is going to provide
complete protection. A certain amount of educated human judgment is
going to be needed and, yes, that is going to require more effort and
understanding than a simple, minimal, list of code points to be
avoided... just as learning to competently use a different language
or writing system does.
In the terms of your note below, is "better than default" useful?
Perhaps, but we need to ask the question, as to whether, by
introducing even more options and, more important, giving people a
good excuse to avoid more focused specifications in cases where those
specifications are likely to be important, it wouldn't cause more
harm than good.
Is that going to be hard for some people? Yes. But routing is hard
and we don't tell people setting up routing procedures "oh, just let
it go, ignore those complex rules about packet formats and their
meaning, put the packets out there and hope for the best". Nor do
we consider simple substitution cyphers appropriate where
confidentiality is needed even though they are lots easier for
non-experts to understand. Interoperability is often hard. The IETF
has usually decided it is worth it.
The above is the argument for the more complex procedures of those
other specs and even for easier measures like doing case folding and
normalization on string comparison when those are appropriate and
used with the understanding that there are edge cases and that they
are not complete solutions to all possible issues. It that sounds
like I'm skeptical whether, in actual practice, there are such things
as "opaque record keys or URIs or enum values" and whether, if there
were, whether even the level of protection these named subsets would
offer would be worth the trouble, yes, that is probably the case.
While all of the above are arguments against publishing this document
in its present form, there is another way to look at the problem. It
is the way that, of necessity, programming languages and some data
representation languages look at things. XML, for example, cannot
have a PRECIS-style set of restrictions on the characters that can be
represented. If it did, we wouldn't be able to write an RFC that
described and illustrated some of the problem cases except, maybe, as
images. Similarly, for the network, we could adopt something of a
"recipient (or string user, interpreter, etc.) beware" model in which
the only job of a network protocol or data structure is simply to
ensure that whatever goes in comes out again. As the I-D more or
less points out, where Unicode is involved that requires more than
simply storing (or sending) the bits. It involves avoiding a
collection of invalid code points and sequences at the coding level
because, if they are stored and then interpreted by something other
than the originating system(s), something odd might reasonably be
expected to happen. Viewed from that perspective, the I-D does a
fairly good job of identifying and explaining the most important
problem cases.
That distinction could be described as the difference between strings
that are appropriate for processing and strings that are only
appropriate for transmission. In that context, I'm not sure what a
"container format" is and whether that help. My comment about too
many standards may be part of an argument against people inventing
their own rules on a per-protocol, or similar, basis even if we could
magically make that easy.
While I will never, personally, be a big fan of "recipient beware" (a
generalization of "buyer beware"), if the focus is on what can be
transmitted or stored safely with the assumption that the specific
language and script issues with which PRECIS and many W3C specs are
concerned are Someone Else's Problem, then a modified version of the
document probably has value. What modifications would that require?
* Make it very clear that this spec is about transmission and storage
and that it does not address the topics and issues needed to _use_
the character strings (especially if they might be treated as
identifiers) or even what should be stored or transmitted, only what
is need to safely transmit and store them.
* Get rid of the PRECIS profiles. PRECIS addresses a different
problem and the main arguments against these subsets as PRECIS
profiles are that they are not consistent with solutions to that
problem as well as violating the direction against addition of new
profiles without strong justification (and, I would add,
justification in the PRECIS context). Saying something about the
difference between the PRECIS (e.g.) topic area and this spec would
help answer the question of why this spec is useful. That does not
mean you should not mention PRECIS, nor that we should avoid
considering a reference to this spec for mention in a future PRECIS
revision or update. Indeed, while it would strike me as a bit
strange, I would not have a strong objection to this document
updating PRECIS to specify the cross-reference.
* Put in an explicit warning that these subsets/ profiles may not be
sufficient if strings are going to be compared or matched to others
and perhaps not even if they are to be displayed if having the reader
see what the originator intended is important... unless information
is transmitted along with them that permits the recipient system or
reader to sort those things out. The latter is important and may
deserve mention: I don't think it is where you want to go but, if the
document specified that any string transmitted using one of its
subsets/ profiles must be transmitted along with language, locale,
and directionality information, it would at least reduce the
perception of risk.
* Include some pointers, starting with one to PRECIS, about more
processing-oriented specs for those that need them.
* Rethink having three subsets rather than one and/or explain better
why one would want to choose any but the most restrictive. The fact
that some programming, data structure, or specification language
allows things is not good enough: for Internet purposes, they might
just be wrong.
* For private use codepoints, if the conclusion is to retain them as
allowed, add some text strongly suggesting that any protocol (or
"container") that allows such codepoints make provision for
identifying the agreement that specifies their interpretation. I
could even see an IANA registry of profiles for private-use code
point collections unless those, somehow, needed to be secret.
Finally and FWIW, I don't buy "they don't seem to be adopting PRECIS"
as a helpful argument in this case because an equally good
explanation for that behavior is that we have not done a good enough
job of explaining why PRECIS (or specs with even more specific
detail) are important. Instead, we have said things that sound to
the naive or lazy reader like "it is ok to use whatever XML, JSON,
etc., allow" with the implied "why go to the extra work". If the
IESG were to require --as Section 6 of RFC 2277 (BCP 18) specifies --
that every specification involving the use of non-ASCII characters
contain an "Internationalization Considerations" section and, if the
spec did not point to PRECIS, that it explained why not and,
presumably, what was being done instead, we would probably be seeing
a different adoption pattern or would understand why not.
--On Thursday, February 13, 2025 12:46 -0800 Tim Bray
<tbray@xxxxxxxxxxxxxx> wrote:
Not going to go point-by-point through John's lengthy missive
and the following correspondence, but the following are worth
addressing:
1. Why Unichars exists
People make new container formats. When they do that, they have to
decide what their character repertoire is. For a general-purpose
low-level container format, PRECIS is a hard sell; it's
complicated, 43 pages, many references, full of discussions of
history, and contains a bunch of policy choices that people are
reluctant to adopt without understanding. [Example: Unassigned code
points are disallowed.] I'm not saying it's bad, just that
empirically, people devising new container formats tend not to
adopt it.
Other people build new protocols around JSON or CBOR or whatever
and, unless they think about it, end up willing to accept those
format's default character subsets, which is bad. However, just
like people devising formats, empirically they don't seem to be
adopting PRECIS.
So, Unichars tries to do two things: First, Offer a succinct
self-contained discussion of the issues with Unicode code points
and why some are "problematic". Second, provide
better-than-default candidate subsets which are easy to understand,
specify, and support, and avoid problematic code points.
As John points out, there are many other discussions of these
issues from various places. But I don't think there's anything
else around that offers these two benefits, and the process of
developing Unichars has not weakened that opinion.
2. Unichars PRECIS profiles
I actually do see values in the PRECIS profiles. Specifically, two
things.
First, the 3 Unichars subsets aim to cover the basic things that
people who are designing low-level container formats and protocols
should think about. In that context there are going to be cases
where they quite reasonably won't want to impose PRECIS's two
existing protocols on their producers or consumers, and would be
glad to use Unichars Assignables or some such. But, in those cases,
I think Unichars ought to at least mention the existence of PRECIS
- if they decide that the content of some particular field has a
high chance of being displayed for consumption by humans, they
should really at least look at PRECIS to inform their decisions
about what to do.
The second reason is the mirror image. Someone who hears that PRECIS
constitutes the IETF rules for Unicode, and realize that some
fields are just opaque record keys or URIs or enum values, and
question why they need the cost and complexity of enforcing PRECIS,
it'd be nice if there were a pointer to Unichars in the registry
so they know there's a simpler option.
So on balance I'd keep the registrations. But if I'm in the
minority on this one I will cheerfully yield without much further
argument.
3. Private Use areas
I searched the correspondence and couldn't find the discussion
behind this one. My recollection is that someone argued strongly
that Unicode says these code points *are considered to be assigned*
(I checked, and indeed it does say that) and that there might well
be scenarios where they are used as intended as part of the
protocol definition, so they shouldn't be seen as problematic.
Once again, this is not a hill I would die on; if consensus is that
PUA code points should be classified as "problematic", OK.
- T
