Should the --encoding argument to log/show commands make any guarantees about their output?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I was surprised to see that the output of

    git log --encoding=utf-8 "--format=format:%b"

can contain byte sequences that are invalid in UTF-8. Note: I am using git 2.1.4 and the %b format specifier represents the commit message body.

I have seen this with the Linux git repository and the following test:

    git log --encoding=utf-8 "--format=format:%b" | python2 -c \
        'import sys; [l.decode("utf-8") for l in sys.stdin]'

Soon enough errors like this appears:

    'utf8' codec can't decode byte 0xf6 in position 19

The help message to the --encoding argument reads:

The commit objects record the encoding used for the log message in
their encoding header; this option can be used to tell the command to
re-code the commit log message in the encoding preferred by the user

I realize that this message does not give any guarantee about the output of the command, in the sense that --encoding=utf-8 produces valid UTF-8 data in all cases.

However, I wonder what --encoding precisely does and if it has the behavior most users would expect.

Let me describe what I think it currently does:

The program attempts to re-code a log message, so it follows the chain

	raw input -> unicode -> raw output

For the first step, knowledge about the input encoding is required. This is retrieved from the encoding header of the commit object if present or (from the docs) "lack of this header implies that the commit log message is encoded in UTF-8." If this step fails (if the entry contains a byte sequence that is invalid in the specified/assumed input codec), the procedure is aborted and the data is dumped as is (obviously without applying the requested output encoding).

Is that correct?

From my point of view the most natural abstraction of a log *message* is *text*, not bytes. The same is true for author names. If I want to build a tool chain on top of log/show, this usually means that I want to work with text information. Hence, I want to retrieve text (a sequence of code points) from git show/log. Text must be transported in encoded form, sure, but it must not contain byte sequences that are invalid in this codec. Because otherwise it's just not text anymore.

Hence, from my point of view, the rational that git show/log should be able to output *text* information means that they should not emit byte sequences that are invalid in the codec specified via the --encoding argument. In the current situation, the work of dealing with invalid byte sequences is just outsourced to software further below in the tool chain (at some point a replacement character � should be displayed to the user instead of the invalid raw bytes).

I am not entirely sure where this discussion should lead to. However, I think that if the behavior of the software will not be changed, then the documentation for the --encoding option should be more precise and clarify what actually happens behind the scenes. What do you think?


Cheers,


Jan-Philip Gehrcke


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]