| Hi all we have started to use CephFS as the backing storage for OpenStack VM instance images. We have one pool (backed by SSDs) in our CephCluster that is exposed via CephFS to the different physical hosts on our machine. The problem we see, is that the different hosts have different views on the filesystem (i.e. they see different amount of files). See an example listing at the end - this lists the virtual disks that each server sees (h0, h1 etc are the name of the servers) All servers mount the same filesystem. Needless to say, that we are a bit worried… Any ideas? cheers jc root@ineri ~$ ndo all_nodes grep instances /etc/fstab h0 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2 h1 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...g==roo,noatime 0 2 h3 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2 h5 [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2 We have upgraded theses servers to the 3.10.10 kernel root@ineri ~$ ndo all_nodes uname -a h0 Linux h0 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux h1 Linux h1 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux h3 Linux h3 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux h5 Linux h5 3.10.10-031010-generic #201308291422 SMP Thu Aug 29 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux and are running ceph 61.7 root@ineri ~$ ndo all_nodes ceph --version h0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff) h1 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff) h2 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff) h3 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff) h4 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff) h5 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff) mds is running on two other nodes, these are on 61.8 root@ineri ~$ ceph --version ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b) root@myoko:~# ceph --version ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b) --- cut --- root@ineri ~$ ndo all_nodes ls -l /mnt/instances/instance-*/disk h0 { -rw-r--r-- 1 root root 4050518016 Aug 30 11:23 /mnt/instances/instance-00000053/disk -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk -rw-r--r-- 1 libvirt-qemu kvm 0 Jan 1 1970 /mnt/instances/instance-0000005e/disk -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk -rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk -rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk -rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk -rw-r--r-- 1 root root 1503854592 Sep 5 18:45 /mnt/instances/instance-0000075f/disk -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk} h1 { -rw-r--r-- 1 root root 4050518016 Aug 30 11:23 /mnt/instances/instance-00000053/disk -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk -rw-r--r-- 1 libvirt-qemu kvm 0 Jan 1 1970 /mnt/instances/instance-0000005e/disk -rw-r--r-- 1 libvirt-qemu kvm 197120 Apr 8 14:25 /mnt/instances/instance-0000006e/disk -rw-r--r-- 1 root root 1545994240 Sep 5 23:25 /mnt/instances/instance-00000070/disk -rw-r--r-- 1 root root 603979776 Sep 5 23:22 /mnt/instances/instance-00000073/disk -rw-r--r-- 1 root root 429654016 Jun 18 09:16 /mnt/instances/instance-000000a4/disk -rw-r--r-- 1 root root 1985019904 Aug 29 14:41 /mnt/instances/instance-000000a7/disk -rw-r--r-- 1 nova nova 197120 May 22 23:28 /mnt/instances/instance-000000c7/disk -rw-r--r-- 1 nova nova 1048576 May 24 14:11 /mnt/instances/instance-000000c8/disk -rw-r--r-- 1 nova nova 1048576 May 24 18:33 /mnt/instances/instance-000000c9/disk -rw-r--r-- 1 nova nova 1048576 May 24 18:35 /mnt/instances/instance-000000ca/disk -rw-r--r-- 1 nova nova 1048576 May 24 18:46 /mnt/instances/instance-000000cb/disk -rw-r--r-- 1 nova nova 1048576 May 24 19:14 /mnt/instances/instance-000000cc/disk -rw-r--r-- 1 nova nova 197120 May 27 15:19 /mnt/instances/instance-000000cd/disk -rw-r--r-- 1 nova nova 197120 May 27 15:29 /mnt/instances/instance-000000ce/disk -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk -rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk -rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk -rw-r--r-- 1 nova nova 221380608 Sep 6 16:06 /mnt/instances/instance-000003ee/disk -rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk} h3 { -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk -rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk -rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk -rw-r--r-- 1 root root 2193424896 Sep 10 16:47 /mnt/instances/instance-00000193/disk -rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk -rw-r--r-- 1 libvirt-qemu kvm 383909888 Sep 13 02:43 /mnt/instances/instance-0000074e/disk -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk} h5 { -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54 /mnt/instances/instance-00000059/disk -rw-r--r-- 1 nova nova 197120 May 22 23:28 /mnt/instances/instance-000000c7/disk -rw-r--r-- 1 nova nova 1048576 May 24 14:11 /mnt/instances/instance-000000c8/disk -rw-r--r-- 1 nova nova 1048576 May 24 18:33 /mnt/instances/instance-000000c9/disk -rw-r--r-- 1 nova nova 1048576 May 24 18:35 /mnt/instances/instance-000000ca/disk -rw-r--r-- 1 nova nova 197120 May 27 15:19 /mnt/instances/instance-000000cd/disk -rw-r--r-- 1 nova nova 197120 May 27 15:29 /mnt/instances/instance-000000ce/disk -rw-r--r-- 1 nova nova 197120 May 27 15:38 /mnt/instances/instance-000000cf/disk -rw-r--r-- 1 nova nova 1048576 May 27 16:36 /mnt/instances/instance-000000d1/disk -rw-r--r-- 1 nova nova 1048576 May 27 16:36 /mnt/instances/instance-000000d2/disk -rw-r--r-- 1 nova nova 1048576 May 27 16:36 /mnt/instances/instance-000000d5/disk -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54 /mnt/instances/instance-000000ec/disk -rw-r--r-- 1 root root 1245773824 Sep 13 08:54 /mnt/instances/instance-00000133/disk -rw-r--r-- 1 root root 414580736 Sep 13 08:54 /mnt/instances/instance-0000013d/disk -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54 /mnt/instances/instance-0000016a/disk -rw-r--r-- 1 nova nova 221380608 Sep 6 16:06 /mnt/instances/instance-000003ee/disk -rw-r--r-- 1 root root 4248043520 Sep 13 08:54 /mnt/instances/instance-0000074a/disk -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54 /mnt/instances/instance-0000074f/disk -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54 /mnt/instances/instance-0000076b/disk -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54 /mnt/instances/instance-0000077d/disk -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54 /mnt/instances/instance-00000792/disk} --- cut --- -- SWITCH Jens-Christian Fischer, Peta Solutions Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 15 71 jens-christian.fischer@xxxxxxxxx |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
