Em Fri, 7 May 2021 10:35:27 +0200 Michal Suchánek <msuchanek@xxxxxxx> escreveu: > On Fri, May 07, 2021 at 10:04:35AM +0200, Mauro Carvalho Chehab wrote: > > Em Fri, 7 May 2021 08:39:24 +0200 > > Mauro Carvalho Chehab <mchehab@xxxxxxxxxx> escreveu: > > > > > Em Thu, 6 May 2021 14:21:01 -0700 > > > Randy Dunlap <rdunlap@xxxxxxxxxxxxx> escreveu: > > > > > > > On 5/6/21 11:08 AM, Matthew Wilcox wrote: > > > > > On Thu, May 06, 2021 at 10:57:53AM -0700, Randy Dunlap wrote: > > > > >> I have been going thru some of the Documentation/ files... > > > > >> > > > > >> Why do several of the files begin with > > > > >> (hex) ef bb bf followed by "==================" > > > > >> for a heading, instead of just "===================". > > > > >> See e.g. Documentation/timers/no_hz.rst. > > > > > > No idea! It seems that the text editor I used on that time added > > > it for whatever reason. > > > > > > > > > > > > > 00000000 ef bb bf 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d |...=============| > > > > > > > > > > ef bb bf is utf8 for 0b1111'111011'111111 = 0xFEFF which is the > > > > > https://en.wikipedia.org/wiki/Byte_order_mark > > > > > > > > > > We should delete it. > > > > > > > > > > > > > OK, thanks, I have started on that. > > > > > > > > > > > > Just another question: ("inquiring minds want to know") > > > > > > > > Why is/are some docs using U+2217 '*' instead of ASCII '*'? > > > > E.g., Documentation/block/cdrom-standard.rst. > > > > > > The cdrom doc is a very special case: it was originally written in LaTeX. > > > I don't remember any other document in LaTeX inside the Kernel docs during > > > the conversions I made. See: > > > e327cfcb2542 ("docs: cdrom-standard.tex: convert from LaTeX to ReST") > > > > > > In order to convert it to .rst, I used some tool to first turn it > > > into plain text (probably LaTeX, but I don't remember anymore), and then > > > I manually reviewed the entire file, adding ReST tags where needed. > > > > > > I didn't realize that utf-8 chars were used instead of normal ASCII chars, > > > as both appear the same when editing it[1]. > > > > > > [1] I use Fedora here. Fedora changed the default charset to utf-8 a long > > > time ago. > > > > > > Anyway, we should be able of get rid of weird UTF-8 chars from it with: > > > > > > $ iconv -f utf-8 -t ascii//TRANSLIT Documentation/cdrom/cdrom-standard.rst > > > > > > I'll prepare a patch fixing it. Some care should be taken, however, as > > > it has two places where UTF-8 chars should be used[2]. > > > > > > [2] There are two German person names that use UTF-8 chars: > > > - 'o' + umlat; > > > - a LATIN SMALL LETTER SHARP S (Eszett) > > > > Btw, I did a quick check here: excluding translations, there are 182 > > files with UTF-8 chars at next-20210429. It seems that most of them > > are on files that got converted from DocBook and html. > > > > Several of them are valid ones: the ones used on names > > (like Günther, Alcôve, ...). > > > 2. Some UTF-8 symbols, like: > > > > - ® > > - ™ > > - ² - used mainly for I²C > > - … > > - ⬍ ↑ ↓ > > - µs - used for microsseconds > > > 3. There are couple of places which uses UTF-8 graphic characters, like: > > > > /sys/devices/system/edac/ > > ├── mc > > │ ├── mc0 > > │ │ ├── ce_count > > │ │ ├── ce_noinfo_count > > > I'm preparing a patchset to address the UTF-8 issues on the top of > > today's next, but before posting, it seems reasonable to discuss > > what to do with the above cases. Comments? > > So the bottom line is that UTF-8 in the files will stay, and Sphinx > cannot handle UTF-8 when the locale is not UTF-8. Yes. We can reduce the number of UTF-8, but some documents need more chars than ASCII can provide. Btw, probably (almost?) all files under Documentation/translation use UTF-8 charsets, due to obvious reasons. > In the long run it might be nice to fix Sphinx to properly set the > encoding of the files it reads and writes. Agreed. > Or maybe there is some parameter that specifies it? > > For the short term I think it is reasonable to run a python test script > that prints fancy unicode characters before running Sphinx and bail if > the test script fails. > > eg. > echo 'print("↑ᛏ个")' > test.py > python3 test.py Actually, a better workaround could be introduced at conf.py. This file is read/parsed by Sphinx on an early stage. Something could be added there that would detect if the machine's charset is not UTF-8 and either produce a warning before starts building or would change the charset used by python to something that won't crash with utf-8. Thanks, Mauro
