> Am 31.08.2017 um 21:43 schrieb Jonathan Corbet <corbet@xxxxxxx>:
>
> On Thu, 31 Aug 2017 22:21:29 +0300
> Jani Nikula <jani.nikula@xxxxxxxxx> wrote:
>
>> On python3, Popen() universal_newlines=True converts the subprocess
>> stdout to unicode text using a codec based on user preferences. Given
>> LANG indicating ascii and utf-8 stdout from the subprocess, you'd get:
>>
>> WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno
>> ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't
>> decode byte 0xe2 in position 6368: ordinal not in range(128)
>>
>> Fix this by dropping universal_newlines=True and replacing the implicit
>> LANG specific decode with an explicit utf-8 decode. This also gets rid
>> of the annoying conditional code for python 2 vs. 3.
>>
>> Fixes: ba3501859354 ("Documentation/sphinx: fix kernel-doc extension on python3")
>> Reference: http://mid.mail-archive.com/54c23e8e-89c0-5cea-0dcc-e938952c5642@xxxxxxxxxxxxx
>> Reported-and-tested-by: Randy Dunlap <rdunlap@xxxxxxxxxxxxx>
>> Cc: Jonathan Corbet <corbet@xxxxxxx>
>> Cc: Mauro Carvalho Chehab <mchehab@xxxxxxxxxxxxxxxx>
>> Signed-off-by: Jani Nikula <jani.nikula@xxxxxxxxx>
>
> Cool...I go out to run some errands and the problem's fixed! :)
>
> Patch applied, thanks to everybody for figuring this out.
Sorry for my absence on this, I have been on a journey. Just let me
throw my 5 cent behind:
Sometimes I really hate python for crap stuff like
"universal newline support" [4]
Started in Python 2 with universal-newline feature [1]
"""If universal_newlines is True, the file objects stdout and
stderr are opened as text files in universal newlines mode.
Lines may be terminated by any of '\n', the Unix end-of-line convention,
'\r', the old Macintosh convention or '\r\n', the Windows convention. All
of these external representations are seen as '\n' by the Python program."""
First it sounds fine ... "but" read on:
"""Note:
This feature is only available if Python is built with universal newline
support (the default). Also, the newlines attribute of the file objects
stdout, stdin and stderr are not updated by the communicate() method.
"""
Then it comes to Python 3 and after a while -- in Python 3.3 -- crapy
"universal newline" has been revised [2]:
"""Changed in version 3.3: When universal_newlines is True, the
class uses the encoding locale.getpreferredencoding(False) instead
of locale.getpreferredencoding(). ..."""
And here is what locale.getpreferredencoding(do_setlocale=False) says [3]...
"""locale.getpreferredencoding(do_setlocale=True)
Return the encoding used for text data, according to user preferences.
User preferences are expressed differently on different systems, and might
not be available programmatically on some systems, so this function
only returns a guess.
On some systems, it is necessary to invoke setlocale() to obtain the user
preferences, so this function is not thread-safe. If invoking setlocale is
not necessary or desired, do_setlocale should be set to False."""
To summarize:
1. universal-newline is an option which has to be compiled into the interpreter
2. universal-newline is based on a 'guess'
3. the way to get such a 'guess' has been changed in Python 3.3
I was suspiciously in "universal stuff" since the first occurrence.
This confirms me, that it is crap and Jani's patch -- removing it --
is the only and right answer. / Thanks!
-- Markus --
[1] https://docs.python.org/2/library/subprocess.html
[2] https://docs.python.org/3.3/library/subprocess.html#frequently-used-arguments
[3] https://docs.python.org/3/library/locale.html#locale.getpreferredencoding
[4] https://www.python.org/dev/peps/pep-0278/
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
