On Fri, May 14, 2021 at 09:17:30AM -0700, Darrick J. Wong wrote: > On Fri, May 14, 2021 at 09:19:45AM +1000, Dave Chinner wrote: > > On Thu, May 13, 2021 at 11:52:52AM -0700, Darrick J. Wong wrote: > > > On Thu, May 13, 2021 at 07:44:59PM +0200, Jan Kara wrote: > > > > On Wed 12-05-21 08:23:45, Darrick J. Wong wrote: > > > > > On Wed, May 12, 2021 at 03:46:11PM +0200, Jan Kara wrote: > > > > > > +->fallocate implementation must be really careful to maintain page cache > > > > > > +consistency when punching holes or performing other operations that invalidate > > > > > > +page cache contents. Usually the filesystem needs to call > > > > > > +truncate_inode_pages_range() to invalidate relevant range of the page cache. > > > > > > +However the filesystem usually also needs to update its internal (and on disk) > > > > > > +view of file offset -> disk block mapping. Until this update is finished, the > > > > > > +filesystem needs to block page faults and reads from reloading now-stale page > > > > > > +cache contents from the disk. VFS provides mapping->invalidate_lock for this > > > > > > +and acquires it in shared mode in paths loading pages from disk > > > > > > +(filemap_fault(), filemap_read(), readahead paths). The filesystem is > > > > > > +responsible for taking this lock in its fallocate implementation and generally > > > > > > +whenever the page cache contents needs to be invalidated because a block is > > > > > > +moving from under a page. > > > > > > + > > > > > > +->copy_file_range and ->remap_file_range implementations need to serialize > > > > > > +against modifications of file data while the operation is running. For blocking > > > > > > +changes through write(2) and similar operations inode->i_rwsem can be used. For > > > > > > +blocking changes through memory mapping, the filesystem can use > > > > > > +mapping->invalidate_lock provided it also acquires it in its ->page_mkwrite > > > > > > +implementation. > > > > > > > > > > Question: What is the locking order when acquiring the invalidate_lock > > > > > of two different files? Is it the same as i_rwsem (increasing order of > > > > > the struct inode pointer) or is it the same as the XFS MMAPLOCK that is > > > > > being hoisted here (increasing order of i_ino)? > > > > > > > > > > The reason I ask is that remap_file_range has to do that, but I don't > > > > > see any conversions for the xfs_lock_two_inodes(..., MMAPLOCK_EXCL) > > > > > calls in xfs_ilock2_io_mmap in this series. > > > > > > > > Good question. Technically, I don't think there's real need to establish a > > > > single ordering because locks among different filesystems are never going > > > > to be acquired together (effectively each lock type is local per sb and we > > > > are free to define an ordering for each lock type differently). But to > > > > maintain some sanity I guess having the same locking order for doublelock > > > > of i_rwsem and invalidate_lock makes sense. Is there a reason why XFS uses > > > > by-ino ordering? So that we don't have to consider two different orders in > > > > xfs_lock_two_inodes()... > > > > > > I imagine Dave will chime in on this, but I suspect the reason is > > > hysterical raisins^Wreasons. > > > > It's the locking rules that XFS has used pretty much forever. > > Locking by inode number always guarantees the same locking order of > > two inodes in the same filesystem, regardless of the specific > > in-memory instances of the two inodes. > > > > e.g. if we lock based on the inode structure address, in one > > instancex, we could get A -> B, then B gets recycled and > > reallocated, then we get B -> A as the locking order for the same > > two inodes. > > > > That, IMNSHO, is utterly crazy because with non-deterministic inode > > lock ordered like this you can't make consistent locking rules for > > locking the physical inode cluster buffers underlying the inodes in > > the situation where they also need to be locked. > > <nod> That's protected by the ILOCK, correct? > > > We've been down this path before more than a decade ago when the > > powers that be decreed that inode locking order is to be "by > > structure address" rather than inode number, because "inode number > > is not unique across multiple superblocks". > > > > I'm not sure that there is anywhere that locks multiple inodes > > across different superblocks, but here we are again.... > > Hm. Are there situations where one would want to lock multiple > /mappings/ across different superblocks? The remapping code doesn't > allow cross-super operations, so ... pipes and splice, maybe? I don't > remember that code well enough to say for sure. Hmmmm. Doing read IO into a buffer that is mmap()d from another file, and we take a page fault on it inside the read IO path? We're copying from a page in one mapping and taking a fault in another mapping and hence taking the invalidate_lock to populate the page cache for the second mapping... I haven't looked closely enough at where the invalidate_lock is held in the read path to determine if this is an issue, but if it is then it is also a potential deadlock scenario... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
