On Fri, 25 May 2018, Oliver Freyermuth wrote: > Dear Ric, > > I played around a bit - the common denominator seems to be: Moving it > within a directory subtree below a directory for which max_bytes / > max_files quota settings are set, things work fine. Moving it to another > directory tree without quota settings / with different quota settings, > rename() returns EXDEV. Aha, yes, this is the issue. When you set a quota you force subvolume-like behavior. This is done because hard links across this quota boundary won't correctly account for utilization (only one of the file links will accrue usage). The expectation is that quotas are usually set in locations that aren't frequently renamed across. It might be possible to allow rename(2) to proceed in cases where nlink==1, but the behavior will probably seem inconsistent (some files get EXDEV, some don't). sage > > Cheers, Oliver > > > Am 25.05.2018 um 15:18 schrieb Ric Wheeler: > > That seems to be the issue - we need to understand why rename sees them as different. > > > > Ric > > > > > > On Fri, May 25, 2018, 9:15 AM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx>> wrote: > > > > Mhhhm... that's funny, I checked an mv with an strace now. I get: > > --------------------------------------------------------------------------------- > > access("/cephfs/some_folder/file", W_OK) = 0 > > rename("foo", "/cephfs/some_folder/file") = -1 EXDEV (Invalid cross-device link) > > unlink("/cephfs/some_folder/file") = 0 > > lgetxattr("foo", "security.selinux", "system_u:object_r:fusefs_t:s0", 255) = 30 > > --------------------------------------------------------------------------------- > > But I can assure it's only a single filesystem, and a single ceph-fuse client running. > > > > Same happens when using absolute paths. > > > > Cheers, > > Oliver > > > > Am 25.05.2018 um 15:06 schrieb Ric Wheeler: > > > We should look at what mv uses to see if it thinks the directories are on different file systems. > > > > > > If the fstat or whatever it looks at is confused, that might explain it. > > > > > > Ric > > > > > > > > > On Fri, May 25, 2018, 9:04 AM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx> <mailto:freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx>>> wrote: > > > > > > Am 25.05.2018 um 14:57 schrieb Ric Wheeler: > > > > Is this move between directories on the same file system? > > > > > > It is, we only have a single CephFS in use. There's also only a single ceph-fuse client running. > > > > > > What's different, though, are different ACLs set for source and target directory, and owner / group, > > > but I hope that should not matter. > > > > > > All the best, > > > Oliver > > > > > > > Rename as a system call only works within a file system. > > > > > > > > The user space mv command becomes a copy when not the same file system. > > > > > > > > Regards, > > > > > > > > Ric > > > > > > > > > > > > On Fri, May 25, 2018, 8:51 AM John Spray <jspray@xxxxxxxxxx <mailto:jspray@xxxxxxxxxx> <mailto:jspray@xxxxxxxxxx <mailto:jspray@xxxxxxxxxx>> <mailto:jspray@xxxxxxxxxx <mailto:jspray@xxxxxxxxxx> <mailto:jspray@xxxxxxxxxx <mailto:jspray@xxxxxxxxxx>>>> wrote: > > > > > > > > On Fri, May 25, 2018 at 1:10 PM, Oliver Freyermuth > > > > <freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx> <mailto:freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx>> <mailto:freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx> <mailto:freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx>>>> wrote: > > > > > Dear Cephalopodians, > > > > > > > > > > I was wondering why a simple "mv" is taking extraordinarily long on CephFS and must note that, > > > > > at least with the fuse-client (12.2.5) and when moving a file from one directory to another, > > > > > the file appears to be copied first (byte by byte, traffic going through the client?) before the initial file is deleted. > > > > > > > > > > Is this true, or am I missing something? > > > > > > > > A mv should not involve copying a file through the client -- it's > > > > implemented in the MDS as a rename from one location to another. > > > > What's the observation that's making it seem like the data is going > > > > through the client? > > > > > > > > John > > > > > > > > > > > > > > For large files, this might be rather time consuming, > > > > > and we should certainly advise all our users to not move files around needlessly if this is the case. > > > > > > > > > > Cheers, > > > > > Oliver > > > > > > > > > > > > > > > _______________________________________________ > > > > > ceph-users mailing list > > > > > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > >
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
