On Fri, 30 Jul 2021, Miklos Szeredi wrote: > On Fri, 30 Jul 2021 at 09:34, NeilBrown <neilb@xxxxxxx> wrote: > > > But I'm curious about your reference to "some sort of subvolume > > structure that the VFS knows about". Do you have any references, or can > > you suggest a search term I could try? > > Found this: > https://lore.kernel.org/linux-fsdevel/20180508180436.716-1-mfasheh@xxxxxxx/ > Excellent, thanks. Very useful. OK. Time for a third perspective. With its current framing the problem is unsolvable. So it needs to be reframed. By "current framing", I mean that we are trying to get btrfs to behave in a way that meets current user-space expectations. Specially, the expectation that each object in any filesystem can be uniquely identified by a 64bit inode number. btrfs provides functionality which needs more than 64bits. So it simple does not fit. btrfs currently fudges with device numbers to hide the problem. This is at best an incomplete solution, and is already being shown to be insufficient. Therefore we need to change user-space expectations. This has been done before multiple times - often by breaking things and leaving it up to user-space to fix it. My favourite example is that NFSv2 broke the creation of lock files with O_CREAT|O_EXCL. USER-space starting using hard-links to achieve the same result. When NFSv3 added reliable O_CREAT|O_EXCL support, it hardly mattered.... but I digress. It think we need to bite-the-bullet and decide that 64bits is not enough, and in fact no number of bits will ever be enough. overlayfs makes this clear. overlayfs merges multiple filesystems, and so needs strictly more bits to uniquely identify all inodes than any of the filesystems use. Currently it over-loads the high bits and hopes the filesystem doesn't use them. The "obvious" choice for a replacement is the file handle provided by name_to_handle_at() (falling back to st_ino if name_to_handle_at isn't supported by the filesystem). This returns an extensible opaque byte-array. It is *already* more reliable than st_ino. Comparing st_ino is only a reliable way to check if two files are the same if you have both of them open. If you don't, then one of the files might have been deleted and the inode number reused for the other. A filehandle contains a generation number which protects against this. So I think we need to strongly encourage user-space to start using name_to_handle_at() whenever there is a need to test if two things are the same. This frees us to be a little less precise about assuring st_ino is always unique, but only a little. We still want to minimize conflicts and avoid them in common situations. A filehandle typically has some bytes used to locate the inode - "location" - and some to validate it - "generation". In general, st_ino must now be seen as a hash of the "location". It could be a generic hash (xxhash? jhash?) or it could be a careful xor of the bits. For btrfs, the "location" is root.objectid ++ file.objectid. I think the inode should become (file.objectid ^ swab64(root.objectid)). This will provide numbers that are unique until you get very large subvols, and very many subvols. It also ensures that two inodes in the same subvol will be guaranteed to have different st_ino. This will quickly cause problems for overlayfs as it means that if btrfs is used with overlayfs, the top few bits won't be zero. Possibly btrfs could be polite and shift the swab64(root.objectid) down 8 bits to make room. Possible overlayfs should handle this case (top N-bits not all zero), and switch to a generic hash of the inode number (or preferably the filehandle) to (64-N bits). If we convince user-space to use filehandles to compare objects, the NFS problems I initially was trying to address go away. Even before that, if btrfs switches to a hashed (i.e. xor) inode number, then the problems also go away. but they aren't the only problems here. Accessing the fhandle isn't always possible. For example reading /proc/locks reports major:minor:inode-number for each file (This is the major:minor from the superblock, so btrfs internal dev numbers aren't used). The filehandle is simply not available. I think the only way to address this is to create a new file. "/proc/locks2" :-) Similarly the "lock:" lines in /proc/$PID/fdinfo/$FD need to be duplicated as "lock2:" lines with filehandle instead of inode number. Ditto for 'inotify:' lines and possibly others. Similarly /proc/$PID/maps contains the inode number with no fhandle. The situation isn't so bad there as there is a path name, and you can even call name_to_handle_at("/proc/$PID/map_files/$RANGE") to get the fhandle. It might be better to provide a new file though. Next we come to the use of different device numbers in the one btrfs filesystem. I'm of the opinion that this was an unfortunately choice that we need to deprecate. Tools that use fhandle won't need it to differentiate inodes, but there is more to the story than just that need. As has been mentioned, people depend on "du -x" and "find -mount" (aka "-xdev") to stay within a "subvol". We need to provide a clean alternate before discouraging that usage. xfs, ext4, fuse, and f2fs each (can) maintain a "project id" for each inode, which effectively groups inodes into a tree. This is used for project quotas. At one level this is conceptually very similar to the btrfs subtree.root.objectid. It is different in that it is only 32 bits (:-[) and is mapped between user name-spaces like uids and gids. It is similar in that it identifies a group of inodes that are accounted together and are (generally) contiguous in a tree. If we encouraged "du" to have a "--proj" option (-j) which stays within a project, and gave a similar option to find, that could be broadly useful. Then if btrfs provided the subvol objectid as fsx_projid (available in FS_IOC_FSGETXATTR ioctl), then "du --proj" on btrfs would stay in a subvol. Longer term it might make sense to add a 64bit project-id to statx. I don't think it would make sense for btrfs to have a 'project' concept that is different from the "subvolume". It would be cool if "df" could have a "--proj" (or similar) flag so that it would report the usage of a "subtree" (given a path). Unfortunately there isn't really an interface for this. Going through the quota system might be nice, I don't think it would work. Another thought about btrfs device numbers is that, providing inode numbers are (nearly) unique, we don't really need more than 2. A btrfs filesystem could allocate 2 anon device numbers. One would be assigned to the root, and each subvolume would get whichever device number its parent doesn't have. This would stop "du -x" and "find -mount" and similar from crossing into subvols. There could be a mount option to select between "1", "2", and "many" device numbers for a filesystem. - I note that cephfs place games with st_dev too.... I wonder if we can learn anything from that. - audit uses sb->s_dev without asking the filesystem. So it won't handle btrfs correctly. I wonder if there is room for it to use file handles. I accept that I'm proposing some BIG changes here, and they might break things. But btrfs is already broken in various ways. I think we need a goal to work towards which will eventually remove all breakage and still have room for expansion. I think that must include: - providing as-unique-as-practical inode numbers across the whole filesystem, and deprecating the internal use of different device numbers. Make it possible to mount without them ASAP, and aim to make that the default eventually. - working with user-space tool/library developers to use name_to_handle_at() to identify inodes, only using st_ino as a fall-back - adding filehandles to various /proc etc files as needed, either duplicating lines or duplicating files. And helping application which use these files to migrate (I would *NOT* change the dev numbers in the current file to report the internal btrfs dev numbers the way that SUSE does. I would prefer that current breakage could be used to motivate developers towards depending instead on fhandles). - exporting subtree (aka subvol) id to user-space, possibly paralleling proj_id in some way, and extending various tools to understand subtrees Who's with me?? NeilBrown
