Re: ceph fs mv does copy, not move

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I don't think so. It is exactly the same location in all tests and it is reproducible.

Why would a move be a copy on some MDSs/OSDs but not others?

Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Marc <Marc@xxxxxxxxxxxxxxxxx>
Sent: 22 June 2021 13:53:25
To: Frank Schilder; 'ceph-users@xxxxxxx'
Subject: RE: ceph fs mv does copy, not move

Could this not be related to the mds and different osd's being used?



> -----Original Message-----
> From: Frank Schilder <frans@xxxxxx>
> Sent: Tuesday, 22 June 2021 13:25
> To: ceph-users@xxxxxxx
> Subject:  Re: ceph fs mv does copy, not move
>
> I get really strange timings depending on kernel version; see below. Did
> the patch of the kernel client get lost? The only difference between
> gnosis and smb01 is that gnosis is physical and smb01 is a KVM. Both
> have direct access to the client network and use the respective kernel
> clients.
>
> Timings of moving a 3.2G file around, crossing quota realms:
>
> [root@gnosis ~]# uname -r
> 5.9.9-1.el7.elrepo.x86_64
>
> [root@gnosis ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m0.013s
> user    0m0.000s
> sys     0m0.001s
>
> [root@gnosis ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/
> real    0m0.008s
> user    0m0.000s
> sys     0m0.001s
>
> [root@gnosis ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m16.807s
> user    0m0.000s
> sys     0m2.150s
>
> [root@gnosis ~]# time \rm /mnt/cephfs/B/testfile.bin
> real    0m0.004s
> user    0m0.000s
> sys     0m0.001s
>
>
>
> [root@smb01 ~]# uname -r
> 5.12.12-1.el7.elrepo.x86_64
>
> [root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m8.545s
> user    0m0.004s
> sys     0m1.999s
>
> [root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/
> real    0m2.986s
> user    0m0.003s
> sys     0m1.644s
>
> [root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m3.564s
> user    0m0.003s
> sys     0m1.671s
>
> [root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin
> real    0m0.022s
> user    0m0.001s
> sys     0m0.000s
>
>
>
> [root@smb01 ~]# uname -r
> 5.4.127-1.el7.elrepo.x86_64
>
> [root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m17.268s
> user    0m0.000s
> sys     0m2.010s
>
> [root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/
> real    0m3.404s
> user    0m0.002s
> sys     0m1.635s
>
> [root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m3.569s
> user    0m0.002s
> sys     0m1.825s
>
> [root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin
> real    0m0.031s
> user    0m0.001s
> sys     0m0.001s
>
>
>
> [root@smb01 ~]# uname -r
> 3.10.0-1160.31.1.el7.x86_64
>
> [root@smb01 ~]# time mv /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m25.695s
> user    0m0.003s
> sys     0m1.936s
>
> [root@smb01 ~]# time mv /mnt/cephfs/B/testfile.bin /mnt/cephfs/A/
> real    0m3.173s
> user    0m0.001s
> sys     0m2.302s
>
> [root@smb01 ~]# time cp /mnt/cephfs/A/testfile.bin /mnt/cephfs/B/
> real    0m2.674s
> user    0m0.004s
> sys     0m2.057s
>
> [root@smb01 ~]# time \rm /mnt/cephfs/B/testfile.bin
> real    0m0.016s
> user    0m0.000s
> sys     0m0.001s
>
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Frank Schilder <frans@xxxxxx>
> Sent: 22 June 2021 11:15:06
> To: ceph-users@xxxxxxx
> Subject:  ceph fs mv does copy, not move
>
> Dear all,
>
> some time ago I reported that the kernel client resorts to a copy
> instead of move when moving a file across quota domains. I was told that
> the fuse client does not have this problem. If enough space is
> available, a move should be a move, not a copy.
>
> Today, I tried to move a large file across quota domains testing botn,
> the kernel- and the fuse client. Both still resort to a copy even though
> this issue was addressed quite a while ago
> (https://lists.ceph.io/hyperkitty/list/ceph-
> users@xxxxxxx/thread/44AEIHNEGKV4VGCARRTARGFZ264CR4T7/#XY7ZCE3KWHI4QSUNZ
> HDWL3QZQFOHXRQW). The versions I'm using are (CentOS 7)
>
> # yum list installed | grep ceph-fuse
> ceph-fuse.x86_64                      2:13.2.10-0.el7
> @ceph
>
> # uname -r
> 3.10.0-1160.31.1.el7.x86_64
>
> Any suggestions how to get this to work? I have to move directories
> containing 100+ TB.
>
> Many thanks,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux