On 2013/11/19 19:10, t.p. wrote:
----- Original Message -----
From: "Martin J. Dürst"<duerst@xxxxxxxxxxxxxxx>
For UTF-8, the BOM is not a Byte Order Mark, because such a mark isn't
necessary at all. It may serve as a signature, but is not necessary,
and
in some circumstances counterproductive.
Martin
We had a similar discussion with syslog back in 2005, the issue being
that UTF-8 was new and different and how to tell whether it was being
used or not, and what made it into RFC5424 was
" If a syslog application encodes MSG in UTF-8, the string MUST start
with the Unicode byte order mask (BOM), which for UTF-8 is ABNF
%xEF.BB.BF. "
which remains a MUST to this day. There are no relevant Errata.
Tom Petch
This is something that seems to have made quite a lot of sense for
syslog. I can understand that if before 2005, syslog was used with
legacy encodings (iso-8859-1, Shift_JIS and similar), and there was
otherwise no easy way to label the UTF-8 strings.
But another solution (for syslog, that is) would also have been
possible. As John already pointed out, UTF-8 is very easy to detect
heuristically: If a byte sequence follows the UTF-8 byte pattern, it's
most definitely UTF-8 and not something else. For more background,
please see http://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf, where
that idea came up first.
As for JSON, it doesn't have the problem of legacy encodings. JSON by
definition is encoded in an Unicode encoding form, and it's easy to
distinguish these because of the restrictions on character sequences in
JSON. And this can be done without a BOM (or with a BOM).
What's most important now is to know what receivers actually accept. We
are not in a design phase, we are just updating the definition of JSON
and making sure we fix problems if there are problems, but we have to
use the installed base for the main guidance, not other protocols or
formats.
Regards, Martin.