On Monday 13 April 2020 08:13:45 Kohada.Tetsuhiro@xxxxxxxxxxxxxxxxxxxxxxxxxxx wrote: > > On Wednesday 08 April 2020 03:59:06 Kohada.Tetsuhiro@xxxxxxxxxxxxxxxxxxxxxxxxxxx wrote: > > > > So partial_name_hash() like I used it in this patch series is enough? > > > > > > I think partial_name_hash() is enough for 8/16/21bit characters. > > > > Great! > > > > Al, could you please take this patch series? > > I think it's good. > > > > > Another point about the discrimination of 21bit characters: > > > I think that checking in exfat_toupper () can be more simplified. > > > > > > ex: return a < PLANE_SIZE && sbi->vol_utbl[a] ? sbi->vol_utbl[a] : a; > > > > I was thinking about it, but it needs more refactoring. Currently > > exfat_toupper() is used on other places for UTF-16 (u16 array) and therefore it cannot be extended to take more then 16 > > bit value. > > I’m also a little worried that exfat_toupper() is designed for only utf16. > Currently, it is converting from utf8 to utf32 in some places, and from utf8 to utf16 in others. > Another way would be to unify to utf16. > > > But I agree that this is another step which can be improved. > > Yes. There are two problems with it: We do not know how code points above U+FFFF could be converted to upper case. Basically from exfat specification can be deduced it only for U+0000 .. U+FFFF code points. We asked if we can get answer from MS, but I have not received any response yet. Second problem is that all MS filesystems (vfat, ntfs and exfat) do not use UCS-2 nor UTF-16, but rather some mix between it. Basically any sequence of 16bit values (except those :/<>... vfat chars) is valid, even unpaired surrogate half. So surrogate pair (two 16bit values) represents one unicode code point (as in UTF-16), but one unpaired surrogate half is also valid and represent (invalid) unicode code point of its value. In unicode are not defined code points for values of single / half surrogate. Therefore if we talk about encoding UTF-16 vs UTF-32 we first need to fix a way how to handle those non-representative values in VFS encoding (iocharset=) as UTF-8 is not able to represent it too. One option is to extend UTF-8 to WTF-8 encoding [1] (yes, this is a real and make sense!) and then ideally change exfat_toupper() to UTF-32 without restriction for surrogate pairs values. Btw, same problem with UTF-16 also in vfat, ntfs and also in iso/joliet kernel drivers. [1] - https://simonsapin.github.io/wtf-8/
