Hello! I have looked more deeply at vfat kernel code how is UTF-8 encoding handled and I found out that case-insensitivity is broken, or rather not implemented at all. In fat_fill_super() function is already FIXME comment about this problem /* FIXME: utf8 is using iocharset for upper/lower conversion */ if (sbi->options.isvfat) { sbi->nls_io = load_nls(sbi->options.iocharset); Basically vfat always loads NLS table which is used for strnicmp and tolower functions. When no is specified, then default (iso8859-1) is used. And this applies also when utf8=1 mount option is specified. Also note that kernel's utf8 NLS table does not implement toupper/tolower functions (kernel's NLS API does not support tolower/toupper for non-fixed-8bit encodings, like UTF-8). So when UTF-8 on VFS for VFAT is enabled, then for VFS <--> VFAT conversion are used utf16s_to_utf8s() and utf8s_to_utf16s() functions. But in fat_name_match(), vfat_hashi() and vfat_cmpi() functions is used NLS table (default iso8859-1) with nls_strnicmp() and nls_tolower(). Which means that fat_name_match(), vfat_hashi() and vfat_cmpi() are broken for vfat in UTF-8 mode. I was thinking how to fix it, and the only possible way is to write a uni_tolower() function which takes one Unicode code point and returns lowercase of input's Unicode code point. We cannot do any Unicode normalization as VFAT specification does not say anything about it and MS reference fastfat.sys implementation does not do it neither. So, what would be the best option for implementing that function? unicode_t uni_tolower(unicode_t u); Could a new fs/unicode code help with it? Or it is too tied with NFD normalization and therefore cannot be easily used or extended? New exfat code which is under review and hopefully would be merged, contains own unicode upcase table (as defined by exfat specification) so as exfat is similar to FAT32, maybe reusing it would be a better option? ======================================================================== Proof that vfat in UTF-8 mode is broken and must be fixed: $ mount | grep /mnt/fat /tmp/fat2 on /mnt/fat type vfat (rw,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro) $ ll /mnt/fat/ total 1 drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./ drwxrwxrwt 4 root root 80 Jan 19 22:45 ../ $ touch /mnt/fat/č $ ll /mnt/fat/ total 1 drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./ drwxrwxrwt 4 root root 80 Jan 19 22:45 ../ -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 č* $ touch /mnt/fat/Č $ ll /mnt/fat/ total 1 drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./ drwxrwxrwt 4 root root 80 Jan 19 22:45 ../ -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 Č* -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 č* $ touch /mnt/fat/d $ ll /mnt/fat/ total 1 drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./ drwxrwxrwt 4 root root 80 Jan 19 22:45 ../ -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 d* -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 Č* -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 č* $ touch /mnt/fat/D $ ll /mnt/fat/ total 1 drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./ drwxrwxrwt 4 root root 80 Jan 19 22:45 ../ -rwxr-xr-x 1 pali pali 0 Jan 19 22:51 d* -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 Č* -rwxr-xr-x 1 pali pali 0 Jan 19 22:50 č* As you can see lowercase 'd' and uppercase 'D' are same, but lowercase 'č' and uppercase 'Č' are not same. This is because 'č' is two bytes 0xc4 0x8d sequence and comparing is done by Latin1 table. 0xc4 is in Latin 'Ä' which is already in uppercase. 0x8d is control char so is not changed by tolower/toupper function. Bigger problem can be with U+C9FF code point. In UTF-8 it is encoded as bytes 0xe3 0xa7 0xbf (in Latin1 㧿). If you convert it by Latin1 upper case table you get ç¿ (bytes 0xc3 0xa7 0xbf). First two bytes is valid UTF-8 sequence for character ç = U+00E7. Therefore U+C9FF and U+00E7 may be treated in some cases as same character (when comparing just prefixes), difference only in upper case, which is fully wrong. -- Pali Rohár pali.rohar@xxxxxxxxx
Attachment:
signature.asc
Description: PGP signature