On Sat, Apr 06, 2024 at 11:15:36PM +0800, HAN Yuwei wrote: > > Sorry, I am not very familiar with Unicode nor kernel. Correct me if wrong. > > As to what I have read, kernel seems like using NFD when processing all > UTF-8 related string. > If fs is using these helper function, then I can be sure kernel is applying > NFD to every UTF-8 filenames. > But I can't find any references to these helper function on Github mirror, > how are they used by fs code? For the most part, the kernel's file stysem code doesn't do anything special for Unicode. The exception is that the ext4 and f2fs file systems can have an optional feature which is mostly only used by Android systems to support case insensitive lookups. This is called the "casefold" feature, which is not enabled by default by most desktop or server systems. The casefold feature was developed because Android has a requirement to support case-insensitive lookups, and it had to support Unicode character sets (for example, XFS has support for case insensitive lookups back from the Irix days, but it only supports ASCII), and the alternative to adding support in the kernel for case fodling was this terrible out-of-tree kernel module which use a file system wrapping that was deadlock-prone (which is why the case-folding wrapfs would never be accepted upstream; it was a trash fire). Anyway, I got tired of being asked to debug file system deadlocks which was not the VFS's fault, but was rather caused by this terrible wrapfs kludge used by Android, so I instigated proper case-folding support (ala Windows and MacOS) for the file system types commonly used by Android, namely ext4 and f2fs. So *if* you are using ext4 or f2fs, *and* the file system is specially created with the file system feature flag "casefold", *and* the directory has the casefold flag set, *then* the file system will support case-preserving, case-insensitive lookups. As a side effect of using utf8_strcasecmp, it will also do string comparisons where even if you have not normalized the file banes, so that the filename contained some Unicode character, such as (for example) the NFC form of the Anstrom Sign character (00C5), and you try to look it up using the NFD form of the character (0041 030A), the lookup will succeed, because we use utf8_strcasecmp(). However, this is *only* if case folding is enabled, and in general, it isn't. Aside from this exception (which as I said, is in general only enabled for Android, because most other use cases such as for Desktop, Server, etc. don't really care about MacOS / Windows style case insensitive filename lookups), the Linux VFS in general treats UTF-8 characters as null-terminated byte streams. So the kernel doesn't validate to make sure that a file name is composed of valid UTF-8 code points (e.g., so we don't prohibit the use of Klingon characters which are not recognized by the Unicode consortium), nor does the kernel do any kind of Unicode normalization. So for example, if casefolding is not enabled, 0041 030A and 00C5 will be considered different, and kernel will not force the NFC form (00C5) to the NFD form (0041 030A) or vice versa. Now, because the kernel tries very hard to be blissfully ignorant about the nightmare which is I18N, it is up to the userspace Unicode libraries to normalize strings before passing them to the kernel --- either as data in text files, or as file names. I am very glad that I don't worry about whether the standard normalization form used by the various GNOME, KDE, Unicode, etc., userspace libraries is NFD, NFC, NFKD, or NFKC. That's someone else's problem, and if you don't have casefolding enabled, we will do the filename comparisons using the strcmp() function. Fundamentally, unicode and normalization is a userspace problem, not a kernel problem, except when we don't have a choice (such as for casse insensitive lookups). And there we solve just the smallest part of the problem, and make it userspace's problem for everything else. Cheers, - Ted
