Hi, I just experienced the same behavior as described in http://tracker.ceph.com/issues/10399 I know that Ubuntu 15.04 is not supported, but I thought it might be interesting for you. I have 2 physical hosts, one with lots of disks and one with only the boot/root disk. On both systems I run kvm. I have 3 VMs running ceph mon, 6 VMs each running 2 ceph osds and some other VMs. The mon and osd VMs boot from virtual disks hosted on local disks of the kvm host. The osds access physical disks on the host using RDM. All other VMs boot from rbd hosted on the ceph cluster. Here a list of installed software: root@kvmhost01:~# dpkg -l | grep ceph ii ceph 0.94.2-0ubuntu0.15.04.1 amd64 distributed storage and file system ii ceph-common 0.94.2-0ubuntu0.15.04.1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-fs-common 0.94.2-0ubuntu0.15.04.1 amd64 common utilities to mount and interact with a ceph file system ii ceph-mds 0.94.2-0ubuntu0.15.04.1 amd64 metadata server for the ceph distributed file system ii libcephfs1 0.94.2-0ubuntu0.15.04.1 amd64 Ceph distributed file system client library ii python-cephfs 0.94.2-0ubuntu0.15.04.1 amd64 Python libraries for the Ceph libcephfs library root@kvmhost01:~# dpkg -l | grep -E 'kvm|qemu|libvirt' ii ipxe-qemu 1.0.0+git-20141004.86285d1-1ubuntu3 all PXE boot firmware - ROM images for qemu ii libvirt-bin 1.2.12-0ubuntu14.1 amd64 programs for the libvirt library ii libvirt0 1.2.12-0ubuntu14.1 amd64 library for interfacing with different virtualization systems ii qemu-kvm 1:2.2+dfsg-5expubuntu9.2 amd64 QEMU Full virtualization ii qemu-system-common 1:2.2+dfsg-5expubuntu9.2 amd64 QEMU full system emulation binaries (common files) ii qemu-system-x86 1:2.2+dfsg-5expubuntu9.2 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 1:2.2+dfsg-5expubuntu9.2 amd64 QEMU utilities Here the error logged by libvirt/qemu: osd/osd_types.cc: In function 'bool pg_t::is_split(unsigned int, unsigned int, std::set<pg_t>*) const' thread 7f29c 8415700 time 2015-07-26 14:53:17.118199 osd/osd_types.cc: 459: FAILED assert(m_seed < old_pg_num) ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (()+0x12cc72) [0x7f29d87a3c72] 2: (()+0x20b6f7) [0x7f29d88826f7] 3: (()+0x20b7c0) [0x7f29d88827c0] 4: (()+0x93b81) [0x7f29d870ab81] 5: (()+0xac47f) [0x7f29d872347f] 6: (()+0xacd58) [0x7f29d8723d58] 7: (()+0xae389) [0x7f29d8725389] 8: (()+0xb485f) [0x7f29d872b85f] 9: (()+0x2be40c) [0x7f29d893540c] 10: (()+0x2eee2d) [0x7f29d8965e2d] 11: (()+0x76aa) [0x7f29d42676aa] 12: (clone()+0x6d) [0x7f29d3f9ceed] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' 2015-07-26 12:53:48.440+0000: shutting down and here some info on ceph and what happend: root@kvmhost01:~# ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 13333G 13087G 240G 1.80 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd 0 0 0 2147G 0 libvirt-pool 1 69316M 0.51 2147G 18042 cephfs_data 3 4519M 0.03 2205G 1346 cephfs_metadata 4 8499k 0 2205G 22 .rgw.root 6 848 0 2205G 3 .rgw.control 7 0 0 2205G 8 .rgw 8 706 0 2205G 4 .rgw.gc 9 0 0 2205G 32 .users.uid 10 675 0 2205G 4 .users.email 11 8 0 2205G 1 .users 12 19 0 2205G 2 .rgw.buckets.index 13 0 0 2205G 2 .rgw.buckets 14 7935M 0.06 2205G 4573 root@kvmhost01:~# ceph health HEALTH_WARN too many PGs per OSD (738 > max 300); pool libvirt-pool has too few pgs root@kvmhost01:~# rados df pool name KB objects clones degraded unfound rd rd KB wr wr KB .rgw 1 4 0 0 0 90 70 30 8 .rgw.buckets 8126126 4573 0 0 0 5108 2664 35376 8127189 .rgw.buckets.index 0 2 0 0 0 9968 9962 9934 0 .rgw.control 0 8 0 0 0 0 0 0 0 .rgw.gc 0 32 0 0 0 4928 4896 3576 0 .rgw.root 1 3 0 0 0 65 43 3 3 .users 1 2 0 0 0 0 0 2 2 .users.email 1 1 0 0 0 0 0 1 1 .users.uid 1 4 0 0 0 70 68 60 2 cephfs_data 4628365 1346 0 0 0 4212 159015 4017 4663669 cephfs_metadata 8500 22 0 0 0 76 11168 4079 10581 libvirt-pool 70980227 18042 0 0 0 4544014 124869457 7658948 185182989 rbd 0 0 0 0 0 0 0 0 0 total used 251477028 24039 total avail 13723737372 total space 13981283784 root@kvmhost01:~# ceph osd pool set libvirt-pool pg_num 256 set pool 1 pg_num to 256 root@kvmhost01:~# ceph osd pool set libvirt-pool pgp_num 256 Error EBUSY: currently creating pgs, wait root@chbsskvmtst01:~# ceph osd pool get libvirt-pool pgp_num pgp_num: 128 a few minutes later root@kvmhost01:~# ceph osd pool set libvirt-pool pgp_num 256 set pool 1 pgp_num to 256 root@kvmhost01:~# ceph osd pool get libvirt-pool pgp_num pgp_num: 256 root@kvmhost01:~# ceph osd pool get libvirt-pool pg_num pg_num: 256 root@kvmhost01:~# rados df pool name KB objects clones degraded unfound rd rd KB wr wr KB .rgw 1 4 0 0 0 90 70 30 8 .rgw.buckets 8126126 4573 0 0 0 5108 2664 35376 8127189 .rgw.buckets.index 0 2 0 0 0 9968 9962 9934 0 .rgw.control 0 8 0 0 0 0 0 0 0 .rgw.gc 0 32 0 0 0 4928 4896 3576 0 .rgw.root 1 3 0 0 0 65 43 3 3 .users 1 2 0 0 0 0 0 2 2 .users.email 1 1 0 0 0 0 0 1 1 .users.uid 1 4 0 0 0 70 68 60 2 cephfs_data 4628365 1346 0 0 0 4212 159015 4017 4663669 cephfs_metadata 8500 22 0 0 0 76 11168 4079 10581 libvirt-pool 70923657 18049 0 92 0 2016645 54587609 3461290 81148392 rbd 0 0 0 0 0 0 0 0 0 total used 258125320 24046 total avail 13717090168 total space 13981283784 root@kvmhost01:~# ceph health HEALTH_WARN too many PGs per OSD (770 > max 300); mds cluster is degraded; mds cephmds02 is laggy root@chbsskvmtst01:~# ceph mds stat e33: 1/1/1 up {0=cephmds02=up:replay(laggy or crashed)} root@chbsskvmtst01:~# virsh list --all Id Name State ---------------------------------------------------- 3 cephmon01 running 4 cephmon02 running 11 cephosd01 running 12 cephosd02 running 13 cephosd03 running 14 cephosd04 running 16 cephosd06 running 17 cephosd05 running 20 cephadm01 running - cephmds01 shut off after I started the mds VMs everything went back to normal (except too many PGs per OSD) Let me know if you need more info Bernhard ____________________________________________________________ FREE 3D EARTH SCREENSAVER - Watch the Earth right on your desktop! Check it out at http://www.inbox.com/earth _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |