On Fri, Sep 13, 2013 at 3:09 PM, Jens-Christian Fischer <jens-christian.fischer@xxxxxxxxx> wrote:
Hi allwe have started to use CephFS as the backing storage for OpenStack VM instance images. We have one pool (backed by SSDs) in our CephCluster that is exposed via CephFS to the different physical hosts on our machine.The problem we see, is that the different hosts have different views on the filesystem (i.e. they see different amount of files). See an example listing at the end - this lists the virtual disks that each server sees (h0, h1 etc are the name of the servers)All servers mount the same filesystem. Needless to say, that we are a bit worried…
The bug was introduced in 3.10 kernel, will be fixed in 3.12 kernel by commit 590fb51f1c (vfs: call d_op->d_prune() before unhashing dentry). Sage may backport the fix to 3.11 and 3.10 kernel soon. please use ceph-fuse at present.
Regards
Yan, Zheng
Any ideas?cheersjcroot@ineri ~$ ndo all_nodes grep instances /etc/fstabh0 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2h1 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...g==roo,noatime 0 2h3 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2h5 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2We have upgraded theses servers to the 3.10.10 kernelroot@ineri ~$ ndo all_nodes uname -ah0 Linux h0 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linuxh1 Linux h1 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linuxh3 Linux h3 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linuxh5 Linux h5 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linuxand are running ceph 61.7root@ineri ~$ ndo all_nodes ceph --versionh0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)h1 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)h2 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)h3 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)h4 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)h5 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)mds is running on two other nodes, these are on 61.8root@ineri ~$ ceph --versionceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)root@myoko:~# ceph --versionceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)--- cut ---root@ineri ~$ ndo all_nodes ls -l /mnt/instances/instance-*/diskh0 {-rw-r--r-- 1 root root 4050518016 Aug 30 11:23 /mnt/instances/instance-00000053/disk-rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk-rw-r--r-- 1 libvirt-qemu kvm 0 Jan 1 1970 /mnt/instances/instance-0000005e/disk-rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk-rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk-rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk-rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk-rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk-rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk-rw-r--r-- 1 root root 1503854592 Sep 5 18:45 /mnt/instances/instance-0000075f/disk-rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk-rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk-rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk}
h1 {-rw-r--r-- 1 root root 4050518016 Aug 30 11:23 /mnt/instances/instance-00000053/disk-rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk-rw-r--r-- 1 libvirt-qemu kvm 0 Jan 1 1970 /mnt/instances/instance-0000005e/disk-rw-r--r-- 1 libvirt-qemu kvm 197120 Apr 8 14:25 /mnt/instances/instance-0000006e/disk-rw-r--r-- 1 root root 1545994240 Sep 5 23:25 /mnt/instances/instance-00000070/disk-rw-r--r-- 1 root root 603979776 Sep 5 23:22 /mnt/instances/instance-00000073/disk-rw-r--r-- 1 root root 429654016 Jun 18 09:16 /mnt/instances/instance-000000a4/disk-rw-r--r-- 1 root root 1985019904 Aug 29 14:41 /mnt/instances/instance-000000a7/disk-rw-r--r-- 1 nova nova 197120 May 22 23:28 /mnt/instances/instance-000000c7/disk-rw-r--r-- 1 nova nova 1048576 May 24 14:11 /mnt/instances/instance-000000c8/disk-rw-r--r-- 1 nova nova 1048576 May 24 18:33 /mnt/instances/instance-000000c9/disk-rw-r--r-- 1 nova nova 1048576 May 24 18:35 /mnt/instances/instance-000000ca/disk-rw-r--r-- 1 nova nova 1048576 May 24 18:46 /mnt/instances/instance-000000cb/disk-rw-r--r-- 1 nova nova 1048576 May 24 19:14 /mnt/instances/instance-000000cc/disk-rw-r--r-- 1 nova nova 197120 May 27 15:19 /mnt/instances/instance-000000cd/disk-rw-r--r-- 1 nova nova 197120 May 27 15:29 /mnt/instances/instance-000000ce/disk-rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk-rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk-rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk-rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk-rw-r--r-- 1 nova nova 221380608 Sep 6 16:06 /mnt/instances/instance-000003ee/disk-rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk-rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk-rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk-rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk-rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk}
h3 {-rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk-rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk-rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk-rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk-rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk-rw-r--r-- 1 root root 2193424896 Sep 10 16:47 /mnt/instances/instance-00000193/disk-rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk-rw-r--r-- 1 libvirt-qemu kvm 383909888 Sep 13 02:43 /mnt/instances/instance-0000074e/disk-rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk-rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk-rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk-rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk}
h5 {-rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk-rw-r--r-- 1 nova nova 197120 May 22 23:28 /mnt/instances/instance-000000c7/disk-rw-r--r-- 1 nova nova 1048576 May 24 14:11 /mnt/instances/instance-000000c8/disk-rw-r--r-- 1 nova nova 1048576 May 24 18:33 /mnt/instances/instance-000000c9/disk-rw-r--r-- 1 nova nova 1048576 May 24 18:35 /mnt/instances/instance-000000ca/disk-rw-r--r-- 1 nova nova 197120 May 27 15:19 /mnt/instances/instance-000000cd/disk-rw-r--r-- 1 nova nova 197120 May 27 15:29 /mnt/instances/instance-000000ce/disk-rw-r--r-- 1 nova nova 197120 May 27 15:38 /mnt/instances/instance-000000cf/disk-rw-r--r-- 1 nova nova 1048576 May 27 16:36 /mnt/instances/instance-000000d1/disk-rw-r--r-- 1 nova nova 1048576 May 27 16:36 /mnt/instances/instance-000000d2/disk-rw-r--r-- 1 nova nova 1048576 May 27 16:36 /mnt/instances/instance-000000d5/disk-rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk-rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk-rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk-rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk-rw-r--r-- 1 nova nova 221380608 Sep 6 16:06 /mnt/instances/instance-000003ee/disk-rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk-rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk-rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk-rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk-rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk}
--- cut -----
SWITCH
Jens-Christian Fischer, Peta Solutions
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 15, direct +41 44 268 15 71
jens-christian.fischer@xxxxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
