I don't think so. It is exactly the same location in all tests and it is reproducible. Why would a move be a copy on some MDSs/OSDs but not others? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc@xxxxxxxxxxxxxxxxx> Sent: 22 June 2021 13:53:25 To: Frank Schilder; 'ceph-users@xxxxxxx' Subject: RE: ceph fs mv does copy, not move Could this not be related to the mds and different osd's being used? > -----Original Message----- > From: Frank Schilder <frans@xxxxxx> > Sent: Tuesday, 22 June 2021 13:25 > To: ceph-users@xxxxxxx > Subject: Re: ceph fs mv does copy, not move > > I get really strange timings depending on kernel version; see below. Did > the patch of the kernel client get lost? The only difference between > gnosis and smb01 is that gnosis is physical and smb01 is a KVM. Both > have direct access to the client network and use the respective kernel > clients. > > Timings of moving a 3.2G file around, crossing quota realms: > > [root@gnosis ~]# uname -r > 5.9.9-1.el7.elrepo.x86_64 > > [root@gnosis ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m0.013s > user 0m0.000s > sys 0m0.001s > > [root@gnosis ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ > real 0m0.008s > user 0m0.000s > sys 0m0.001s > > [root@gnosis ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m16.807s > user 0m0.000s > sys 0m2.150s > > [root@gnosis ~]# time \rm /mnt/cephfs/B/testfile.bin > real 0m0.004s > user 0m0.000s > sys 0m0.001s > > > > [root@smb01 ~]# uname -r > 5.12.12-1.el7.elrepo.x86_64 > > [root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m8.545s > user 0m0.004s > sys 0m1.999s > > [root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ > real 0m2.986s > user 0m0.003s > sys 0m1.644s > > [root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m3.564s > user 0m0.003s > sys 0m1.671s > > [root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin > real 0m0.022s > user 0m0.001s > sys 0m0.000s > > > > [root@smb01 ~]# uname -r > 5.4.127-1.el7.elrepo.x86_64 > > [root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m17.268s > user 0m0.000s > sys 0m2.010s > > [root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ > real 0m3.404s > user 0m0.002s > sys 0m1.635s > > [root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m3.569s > user 0m0.002s > sys 0m1.825s > > [root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin > real 0m0.031s > user 0m0.001s > sys 0m0.001s > > > > [root@smb01 ~]# uname -r > 3.10.0-1160.31.1.el7.x86_64 > > [root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m25.695s > user 0m0.003s > sys 0m1.936s > > [root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/ > real 0m3.173s > user 0m0.001s > sys 0m2.302s > > [root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/ > real 0m2.674s > user 0m0.004s > sys 0m2.057s > > [root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin > real 0m0.016s > user 0m0.000s > sys 0m0.001s > > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Frank Schilder <frans@xxxxxx> > Sent: 22 June 2021 11:15:06 > To: ceph-users@xxxxxxx > Subject: ceph fs mv does copy, not move > > Dear all, > > some time ago I reported that the kernel client resorts to a copy > instead of move when moving a file across quota domains. I was told that > the fuse client does not have this problem. If enough space is > available, a move should be a move, not a copy. > > Today, I tried to move a large file across quota domains testing botn, > the kernel- and the fuse client. Both still resort to a copy even though > this issue was addressed quite a while ago > (https://lists.ceph.io/hyperkitty/list/ceph- > users@xxxxxxx/thread/44AEIHNEGKV4VGCARRTARGFZ264CR4T7/#XY7ZCE3KWHI4QSUNZ > HDWL3QZQFOHXRQW). The versions I'm using are (CentOS 7) > > # yum list installed | grep ceph-fuse > ceph-fuse.x86_64 2:13.2.10-0.el7 > @ceph > > # uname -r > 3.10.0-1160.31.1.el7.x86_64 > > Any suggestions how to get this to work? I have to move directories > containing 100+ TB. > > Many thanks, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |