On Tue, Jul 25, 2017 at 07:58:52AM -0500, Eric Blake wrote: > On 07/25/2017 07:48 AM, Daniel P. Berrange wrote: > > Setting LC_ALL=C breaks python apps doing I/O on UTF-8 source > > files. In particular this broke glib-mkenums > > > > Traceback (most recent call last): > > File "/usr/bin/glib-mkenums", line 669, in <module> > > process_file(fname) > > File "/usr/bin/glib-mkenums", line 406, in process_file > > line = curfile.readline() > > File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode > > return codecs.ascii_decode(input, self.errors)[0] > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 849: ordinal not in range(128) > > Eww. Are we sure that's not a bug in python? LC_ALL=C is supposed to be > 8-bit clean (every byte is a character, you can never get an encoding > error), and a superset of 7-bit ascii. The failure makes is sound like > python is incorrectly treating LC_ALL=C as requesting only 7-bit ascii > instead of full 8-bit cleanness. IIUC, whether its 8-bit clean or not is not the problem - even if it is 8-bit clean, it will not enable UTF-8 support so python defaults to the ascii charset, instead of utf-8 causing the failures seen above. > > +++ b/maint.mk > > @@ -117,8 +117,8 @@ news-check-lines-spec ?= 1,10 > > news-check-regexp ?= '^\*.* $(VERSION_REGEXP) \($(today)\)' > > > > # Prevent programs like 'sort' from considering distinct strings to be equal. > > -# Doing it here saves us from having to set LC_ALL elsewhere in this file. > > -export LC_ALL = C > > +# Doing it here saves us from having to set LC_COLLATE elsewhere in this file. > > +export LC_COLLATE = C > > This makes the build non-deterministic on platforms that have exported > LC_ALL. NB it is already non-deterministic because 'maint.mk' is only ever used if you run 'make' from the top-level. So if youre in a sub-dir and run make you never see this setting and thus everything "works" (that made debugging this a total nightmare). Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| _______________________________________________ virt-tools-list mailing list virt-tools-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/virt-tools-list
