On Fri, Apr 24, 2020 at 12:15:16PM +1000, Dave Chinner wrote: > On Thu, Apr 23, 2020 at 04:25:48PM -0700, Ira Weiny wrote: [snap] > > > > + ii> If the file still does not have the desired S_DAX access > > > > + mode, either unmount and remount the filesystem, or close > > > > + the file and use drop_caches. > > > > > > .... don't have permissions to do either of these things... > > > > > > Essentially, you may as well say "reboot the machine" at this point, > > > because it's effectively the same thing from a production workload > > > point of view... > > > > > > Realistically, I'm not sure we should even say "programs must cause > > > eviction", because that's something they cannot do directly without > > > admin privileges nor is it something we want to occur randomly on > > > production machines during production. i.e. this is something that > > > should only be done in scheduled downtime by an administrator, not > > > attempted by applications because DAX isn't immediately available. > > > The admin is in charge here, not the "program". > > > > I agree with everything you say. > > > > But I feel a bit stuck here. Without some type of documentation we are not > > allowing FS_XFLAG_DAX to be changed on a file by the user. Which is what we > > were proposing before and we all disliked. > > For production systems, the admin is the "user" we are taking about. > The program itself shouldn't be choosing the method of file data > access; that's up to the administrator in charge of the system to > set the policy how they want it to be set. > > i.e. there's a difference between the user/admin taking action to > change a data access policy, and the application taking actions to > override the policy that the admin has set. > > What I'm trying to say is that setting/clearing the DAX flags is an > -admin operation-, and part of the consideration of that admin > operation is when the change should take effect. > > i.e. refering to "programs" as if they control the access mode is > entirely the wrong way to be looking at persistent inode flags. They > are an administration policy mechanism that belongs to the data set, > not the application (or "program"). Managing data set storage and > access policy is something administrators do, not the application... Ok. > > > So I feel like we need to say something about getting the inodes evicted. > > perhaps by a 'drop cache' even requested of the admin??? > > > > Maybe this? > > > > > > 4. Programs that require a specific file access mode (DAX or not DAX) > > can do one of the following: > > > > (a) Set the parent directory FS_XFLAG_DAX as needed before file are > > created; or > > > > (b) Have the administrator set the desired behaviour via mount option; or > > > > (c) Set or clear the file's FS_XFLAG_DAX flag as needed and wait for the > > inode to be evicted from memory. > > > > i> the only effective way of ensuring this is to request the admin drop > > the file system caches. > > 4. The DAX policy can be changed via: > > a) Set the parent directory FS_XFLAG_DAX as needed before > files are created > > b) Set the appropriate dax="foo" mount option > > c) Change the FS_XFLAG_DAX on existing regular files and > directories. This has runtime constraints and limitations > that are described in 5) below. > > 5. When changing the DAX policy via toggling the persistent > FS_XFLAG_DAX flag, the change in behaviour for existing regular > files may not occur immediately. If the change must take effect > immediately, the administrator needs to: > > 1. stop the application so there are no active references to > the data set the policy change will affect > 2. evict the data set from kernel caches so it will be > re-instantiated when the application is restarted. This can > be acheived by: > a. drop-caches > b. a filesystem unmount and mount cycle > c. a system reboot > > Hence if DAX access policy changes are required to take immediate > effect, scheduled system-wide downtime will be required to guarantee > the new policy change takes effect when the application is > restarted. > > > > <quote> > > Enabling DAX on xfs > > ------------------- > > > > Summary > > ------- > > > > 1. There exists an in-kernel file access mode flag S_DAX that corresponds to > > the statx flag STATX_ATTR_DAX. See the manpage for statx(2) for details > > about this access mode. > > > > 2. There exists a regular file and directory inode flag FS_XFLAG_DAX. It is > > inherited from the parent directory FS_XFLAG_DAX inode flag at creation > > time. This advisory flag can be set or cleared at any time, but doing so > > does not immediately affect the S_DAX state. > > 2. There exists a persistent flag FS_XFLAG_DAX that can be applied to > regular files and directories. This advisory flag can be set or > cleared at any time, but doing so does not immediately affect the > S_DAX state. Done. > > 3. If the persistent FS_XFLAG_DAX flag is set on a directory, this > flag will be inherited by all regular files and sub directories that > are subsequently created in this directory. Files and subdirectories > that exist at the time this flag is set or cleared on the parent > directory are not modified by this modification of the parent > directory. > Done. > > > > > 3. There exists dax mount options which can override FS_XFLAG_DAX in the > > setting of the S_DAX flag. Given underlying storage which supports DAX the > > following hold. > > > > "-o dax=inode" means "follow FS_XFLAG_DAX" and is the default. > > > > "-o dax=never" means "never set S_DAX, ignore FS_XFLAG_DAX." > > > > "-o dax=always" means "always set S_DAX ignore FS_XFLAG_DAX." > > > > "-o dax" is a legacy option which is an alias for "dax=always". > > This may be removed in the future so "-o dax=always" is > > the preferred method for specifying this behavior. > > > > NOTE: Setting and inheritance affect FS_XFLAG_DAX at all times even when > > the file system is mounted with a dax option. However, in-core inode > > state (S_DAX) will continue to be overridden until the file system is > > s/continue to// Done. > > > remounted with dax=inode and the inode is evicted. > > evicted from kernel memory. Done. > > > > > 4. Programs that require a specific file access mode (DAX or not DAX) > > can do one of the following: > > > > (a) Set the parent directory FS_XFLAG_DAX as needed before file are > > created; or > > > > (b) Have the administrator set the desired behaviour via mount option; or > > > > (c) Set or clear the file's FS_XFLAG_DAX flag as needed and wait for the > > inode to be evicted from memory. > > > > i> the only effective way of ensuring this is to request the admin drop > > the file system caches. > > See my comments above. Done. thanks! > > > > > > > Details > > ------- > > > > There are 2 per-file dax flags. One is a persistent inode setting (FS_XFLAG_DAX) > > and the other is a volatile flag indicating the active state of the feature > > (S_DAX). > > > > FS_XFLAG_DAX is preserved within the file system. This persistent config > > setting can be set, cleared and/or queried using the FS_IOC_FS[GS]ETXATTR ioctl > > (see ioctl_xfs_fsgetxattr(2)) or an utility such as 'xfs_io'. > > 'chattr [-+]x'". > > Stray line. Thanks for the review! V11 should be out soon. Ira > > Cheers, > > Dave. > > -- > Dave Chinner > david@xxxxxxxxxxxxx
