Hello, I'm (almost) a new user of ceph (couple of month). In my university, we start to do some test with ceph a couple of months ago. We have 2 clusters. Each cluster have 100 OSDs on 10 servers : Each server as this setup : CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz Memory : 128GB of Memory OS Storage : 2 x SSD 240GB Intel S3500 DC (raid 1) Journal Storage : 2 x SSD 400GB Intel S3300 DC (no Raid) OSD Disk : 10 x HGST ultrastar-7k6000 6TB Network : 1 x 10Gb/s OS : Ubuntu 14.04 Ceph version : infernalis 9.2.0 One cluster give access to some user through a S3 gateway (service is still in beta). We call this cluster "ceph-beta". One cluster is for our internal need to learn more about ceph. We call this cluster "ceph-test". (those servers will be integrated into the ceph-beta cluster when we will need more space) We have deploy both clusters with the ceph-ansible playbook[1] Journal are raw partitions on SSDs (400GB Intel S3300 DC) with no raid. 5 journals partitions on each SSDs. OSDs disk are format in XFS. 1. https://github.com/ceph/ceph-ansible We have an issue. Some OSDs go down and don't start. It seem to be related to the fsid of the journal partition : > -1> 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal in attachment, the full logs of one of the dead OSDs We had this issue with 2 OSDs on ceph-beta cluster fixed by removing, zapping and readding it. Now, we have the same issue on ceph-test cluster but on 18 OSDs. Now the stats of this cluster > root@icadmin004:~# ceph -s > cluster 4fb4773c-0873-44ad-a65f-269f01bfcff8 > health HEALTH_WARN > 1024 pgs incomplete > 1024 pgs stuck inactive > 1024 pgs stuck unclean > monmap e1: 3 mons at {iccluster003=10.90.37.4:6789/0,iccluster014=10.90.37.15:6789/0,iccluster022=10.90.37.23:6789/0} > election epoch 62, quorum 0,1,2 iccluster003,iccluster014,iccluster022 > osdmap e242: 100 osds: 82 up, 82 in > flags sortbitwise > pgmap v469212: 2304 pgs, 10 pools, 2206 bytes data, 181 objects > 4812 MB used, 447 TB / 447 TB avail > 1280 active+clean > 1024 creating+incomplete We have install this cluster at the begin of February. We did not use that cluster at all even at the begin to troubleshoot an issue with ceph-ansible. We did not push any data neither create pool. What could explain this behaviour ? Thanks for your help Best regards, -- Yoann Moulin EPFL IC-IT
2016-03-03 14:09:00.433074 7efd1a9d5940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4446 2016-03-03 14:09:01.315583 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) 2016-03-03 14:09:01.338328 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2016-03-03 14:09:01.338335 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2016-03-03 14:09:01.338362 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported 2016-03-03 14:09:01.341468 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2016-03-03 14:09:01.341517 7efd1a9d5940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 2016-03-03 14:09:01.411145 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2016-03-03 14:09:01.692400 7efd1a9d5940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 2016-03-03 14:09:01.694251 7efd1a9d5940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7efd1a9d5940 time 2016-03-03 14:09:01.692413 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7efd1a4cbf2b] 2: (()+0x2c2f80) [0x7efd19ed1f80] 3: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a] 5: (FileStore::mount()+0x3b42) [0x7efd1a198a62] 6: (OSD::init()+0x26d) [0x7efd19f51a5d] 7: (main()+0x2954) [0x7efd19ed7474] 8: (__libc_start_main()+0xf5) [0x7efd16d59ec5] 9: (()+0x2f82b7) [0x7efd19f072b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -24> 2016-03-03 14:09:00.430918 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perfcounters_dump hook 0x7efd1e3e0050 -23> 2016-03-03 14:09:00.430937 7efd1a9d5940 5 asok(0x7efd1e460000) register_command 1 hook 0x7efd1e3e0050 -22> 2016-03-03 14:09:00.430938 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perf dump hook 0x7efd1e3e0050 -21> 2016-03-03 14:09:00.430940 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perfcounters_schema hook 0x7efd1e3e0050 -20> 2016-03-03 14:09:00.430941 7efd1a9d5940 5 asok(0x7efd1e460000) register_command 2 hook 0x7efd1e3e0050 -19> 2016-03-03 14:09:00.430943 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perf schema hook 0x7efd1e3e0050 -18> 2016-03-03 14:09:00.430945 7efd1a9d5940 5 asok(0x7efd1e460000) register_command perf reset hook 0x7efd1e3e0050 -17> 2016-03-03 14:09:00.430947 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config show hook 0x7efd1e3e0050 -16> 2016-03-03 14:09:00.430950 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config set hook 0x7efd1e3e0050 -15> 2016-03-03 14:09:00.430954 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config get hook 0x7efd1e3e0050 -14> 2016-03-03 14:09:00.430955 7efd1a9d5940 5 asok(0x7efd1e460000) register_command config diff hook 0x7efd1e3e0050 -13> 2016-03-03 14:09:00.430957 7efd1a9d5940 5 asok(0x7efd1e460000) register_command log flush hook 0x7efd1e3e0050 -12> 2016-03-03 14:09:00.430958 7efd1a9d5940 5 asok(0x7efd1e460000) register_command log dump hook 0x7efd1e3e0050 -11> 2016-03-03 14:09:00.430959 7efd1a9d5940 5 asok(0x7efd1e460000) register_command log reopen hook 0x7efd1e3e0050 -10> 2016-03-03 14:09:00.433016 7efd1a9d5940 0 set uid:gid to 64045:64045 -9> 2016-03-03 14:09:00.433074 7efd1a9d5940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4446 -8> 2016-03-03 14:09:01.315583 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) -7> 2016-03-03 14:09:01.338328 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -6> 2016-03-03 14:09:01.338335 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option -5> 2016-03-03 14:09:01.338362 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported -4> 2016-03-03 14:09:01.341468 7efd1a9d5940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -3> 2016-03-03 14:09:01.341517 7efd1a9d5940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 -2> 2016-03-03 14:09:01.411145 7efd1a9d5940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -1> 2016-03-03 14:09:01.692400 7efd1a9d5940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 0> 2016-03-03 14:09:01.694251 7efd1a9d5940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7efd1a9d5940 time 2016-03-03 14:09:01.692413 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7efd1a4cbf2b] 2: (()+0x2c2f80) [0x7efd19ed1f80] 3: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a] 5: (FileStore::mount()+0x3b42) [0x7efd1a198a62] 6: (OSD::init()+0x26d) [0x7efd19f51a5d] 7: (main()+0x2954) [0x7efd19ed7474] 8: (__libc_start_main()+0xf5) [0x7efd16d59ec5] 9: (()+0x2f82b7) [0x7efd19f072b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:01.696632 7efd1a9d5940 -1 *** Caught signal (Aborted) ** in thread 7efd1a9d5940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7efd1a3df2ca] 2: (()+0x10340) [0x7efd18af3340] 3: (gsignal()+0x39) [0x7efd16d6ecc9] 4: (abort()+0x148) [0x7efd16d720d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd17679535] 6: (()+0x5e6d6) [0x7efd176776d6] 7: (()+0x5d799) [0x7efd17676799] 8: (__gxx_personality_v0()+0x52a) [0x7efd1767734a] 9: (()+0xfff3) [0x7efd1710cff3] 10: (_Unwind_Resume()+0x57) [0x7efd1710d517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7efd1a4cc136] 12: (()+0x2c2f80) [0x7efd19ed1f80] 13: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a] 15: (FileStore::mount()+0x3b42) [0x7efd1a198a62] 16: (OSD::init()+0x26d) [0x7efd19f51a5d] 17: (main()+0x2954) [0x7efd19ed7474] 18: (__libc_start_main()+0xf5) [0x7efd16d59ec5] 19: (()+0x2f82b7) [0x7efd19f072b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2016-03-03 14:09:01.696632 7efd1a9d5940 -1 *** Caught signal (Aborted) ** in thread 7efd1a9d5940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7efd1a3df2ca] 2: (()+0x10340) [0x7efd18af3340] 3: (gsignal()+0x39) [0x7efd16d6ecc9] 4: (abort()+0x148) [0x7efd16d720d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd17679535] 6: (()+0x5e6d6) [0x7efd176776d6] 7: (()+0x5d799) [0x7efd17676799] 8: (__gxx_personality_v0()+0x52a) [0x7efd1767734a] 9: (()+0xfff3) [0x7efd1710cff3] 10: (_Unwind_Resume()+0x57) [0x7efd1710d517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7efd1a4cc136] 12: (()+0x2c2f80) [0x7efd19ed1f80] 13: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a] 15: (FileStore::mount()+0x3b42) [0x7efd1a198a62] 16: (OSD::init()+0x26d) [0x7efd19f51a5d] 17: (main()+0x2954) [0x7efd19ed7474] 18: (__libc_start_main()+0xf5) [0x7efd16d59ec5] 19: (()+0x2f82b7) [0x7efd19f072b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:01.889329 7ff0a521d940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4750 2016-03-03 14:09:01.897263 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) 2016-03-03 14:09:01.897561 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2016-03-03 14:09:01.897565 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2016-03-03 14:09:01.897576 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported 2016-03-03 14:09:01.918907 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2016-03-03 14:09:01.918937 7ff0a521d940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 2016-03-03 14:09:01.965269 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2016-03-03 14:09:01.966401 7ff0a521d940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 2016-03-03 14:09:01.968379 7ff0a521d940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ff0a521d940 time 2016-03-03 14:09:01.966412 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff0a4d13f2b] 2: (()+0x2c2f80) [0x7ff0a4719f80] 3: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a] 5: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62] 6: (OSD::init()+0x26d) [0x7ff0a4799a5d] 7: (main()+0x2954) [0x7ff0a471f474] 8: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5] 9: (()+0x2f82b7) [0x7ff0a474f2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -24> 2016-03-03 14:09:01.887108 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perfcounters_dump hook 0x7ff0a7caa050 -23> 2016-03-03 14:09:01.887136 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command 1 hook 0x7ff0a7caa050 -22> 2016-03-03 14:09:01.887139 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perf dump hook 0x7ff0a7caa050 -21> 2016-03-03 14:09:01.887141 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perfcounters_schema hook 0x7ff0a7caa050 -20> 2016-03-03 14:09:01.887142 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command 2 hook 0x7ff0a7caa050 -19> 2016-03-03 14:09:01.887144 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perf schema hook 0x7ff0a7caa050 -18> 2016-03-03 14:09:01.887147 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command perf reset hook 0x7ff0a7caa050 -17> 2016-03-03 14:09:01.887148 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config show hook 0x7ff0a7caa050 -16> 2016-03-03 14:09:01.887150 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config set hook 0x7ff0a7caa050 -15> 2016-03-03 14:09:01.887153 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config get hook 0x7ff0a7caa050 -14> 2016-03-03 14:09:01.887154 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command config diff hook 0x7ff0a7caa050 -13> 2016-03-03 14:09:01.887159 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command log flush hook 0x7ff0a7caa050 -12> 2016-03-03 14:09:01.887160 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command log dump hook 0x7ff0a7caa050 -11> 2016-03-03 14:09:01.887161 7ff0a521d940 5 asok(0x7ff0a7d2a000) register_command log reopen hook 0x7ff0a7caa050 -10> 2016-03-03 14:09:01.889275 7ff0a521d940 0 set uid:gid to 64045:64045 -9> 2016-03-03 14:09:01.889329 7ff0a521d940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4750 -8> 2016-03-03 14:09:01.897263 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) -7> 2016-03-03 14:09:01.897561 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -6> 2016-03-03 14:09:01.897565 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option -5> 2016-03-03 14:09:01.897576 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported -4> 2016-03-03 14:09:01.918907 7ff0a521d940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -3> 2016-03-03 14:09:01.918937 7ff0a521d940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 -2> 2016-03-03 14:09:01.965269 7ff0a521d940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -1> 2016-03-03 14:09:01.966401 7ff0a521d940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 0> 2016-03-03 14:09:01.968379 7ff0a521d940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ff0a521d940 time 2016-03-03 14:09:01.966412 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff0a4d13f2b] 2: (()+0x2c2f80) [0x7ff0a4719f80] 3: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a] 5: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62] 6: (OSD::init()+0x26d) [0x7ff0a4799a5d] 7: (main()+0x2954) [0x7ff0a471f474] 8: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5] 9: (()+0x2f82b7) [0x7ff0a474f2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:01.970874 7ff0a521d940 -1 *** Caught signal (Aborted) ** in thread 7ff0a521d940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7ff0a4c272ca] 2: (()+0x10340) [0x7ff0a333b340] 3: (gsignal()+0x39) [0x7ff0a15b6cc9] 4: (abort()+0x148) [0x7ff0a15ba0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a1ec1535] 6: (()+0x5e6d6) [0x7ff0a1ebf6d6] 7: (()+0x5d799) [0x7ff0a1ebe799] 8: (__gxx_personality_v0()+0x52a) [0x7ff0a1ebf34a] 9: (()+0xfff3) [0x7ff0a1954ff3] 10: (_Unwind_Resume()+0x57) [0x7ff0a1955517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7ff0a4d14136] 12: (()+0x2c2f80) [0x7ff0a4719f80] 13: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a] 15: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62] 16: (OSD::init()+0x26d) [0x7ff0a4799a5d] 17: (main()+0x2954) [0x7ff0a471f474] 18: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5] 19: (()+0x2f82b7) [0x7ff0a474f2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2016-03-03 14:09:01.970874 7ff0a521d940 -1 *** Caught signal (Aborted) ** in thread 7ff0a521d940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7ff0a4c272ca] 2: (()+0x10340) [0x7ff0a333b340] 3: (gsignal()+0x39) [0x7ff0a15b6cc9] 4: (abort()+0x148) [0x7ff0a15ba0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a1ec1535] 6: (()+0x5e6d6) [0x7ff0a1ebf6d6] 7: (()+0x5d799) [0x7ff0a1ebe799] 8: (__gxx_personality_v0()+0x52a) [0x7ff0a1ebf34a] 9: (()+0xfff3) [0x7ff0a1954ff3] 10: (_Unwind_Resume()+0x57) [0x7ff0a1955517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7ff0a4d14136] 12: (()+0x2c2f80) [0x7ff0a4719f80] 13: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a] 15: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62] 16: (OSD::init()+0x26d) [0x7ff0a4799a5d] 17: (main()+0x2954) [0x7ff0a471f474] 18: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5] 19: (()+0x2f82b7) [0x7ff0a474f2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:05.160375 7fd3b686c940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5719 2016-03-03 14:09:05.169748 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) 2016-03-03 14:09:05.170085 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2016-03-03 14:09:05.170091 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2016-03-03 14:09:05.170107 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported 2016-03-03 14:09:05.192222 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2016-03-03 14:09:05.192299 7fd3b686c940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 2016-03-03 14:09:05.198884 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2016-03-03 14:09:05.200572 7fd3b686c940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 2016-03-03 14:09:05.202443 7fd3b686c940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7fd3b686c940 time 2016-03-03 14:09:05.200585 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fd3b6362f2b] 2: (()+0x2c2f80) [0x7fd3b5d68f80] 3: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a] 5: (FileStore::mount()+0x3b42) [0x7fd3b602fa62] 6: (OSD::init()+0x26d) [0x7fd3b5de8a5d] 7: (main()+0x2954) [0x7fd3b5d6e474] 8: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5] 9: (()+0x2f82b7) [0x7fd3b5d9e2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -24> 2016-03-03 14:09:05.158253 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perfcounters_dump hook 0x7fd3ba926050 -23> 2016-03-03 14:09:05.158272 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command 1 hook 0x7fd3ba926050 -22> 2016-03-03 14:09:05.158274 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perf dump hook 0x7fd3ba926050 -21> 2016-03-03 14:09:05.158276 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perfcounters_schema hook 0x7fd3ba926050 -20> 2016-03-03 14:09:05.158277 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command 2 hook 0x7fd3ba926050 -19> 2016-03-03 14:09:05.158279 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perf schema hook 0x7fd3ba926050 -18> 2016-03-03 14:09:05.158280 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command perf reset hook 0x7fd3ba926050 -17> 2016-03-03 14:09:05.158282 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config show hook 0x7fd3ba926050 -16> 2016-03-03 14:09:05.158283 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config set hook 0x7fd3ba926050 -15> 2016-03-03 14:09:05.158285 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config get hook 0x7fd3ba926050 -14> 2016-03-03 14:09:05.158286 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command config diff hook 0x7fd3ba926050 -13> 2016-03-03 14:09:05.158287 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command log flush hook 0x7fd3ba926050 -12> 2016-03-03 14:09:05.158288 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command log dump hook 0x7fd3ba926050 -11> 2016-03-03 14:09:05.158290 7fd3b686c940 5 asok(0x7fd3ba9a6000) register_command log reopen hook 0x7fd3ba926050 -10> 2016-03-03 14:09:05.160308 7fd3b686c940 0 set uid:gid to 64045:64045 -9> 2016-03-03 14:09:05.160375 7fd3b686c940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5719 -8> 2016-03-03 14:09:05.169748 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) -7> 2016-03-03 14:09:05.170085 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -6> 2016-03-03 14:09:05.170091 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option -5> 2016-03-03 14:09:05.170107 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported -4> 2016-03-03 14:09:05.192222 7fd3b686c940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -3> 2016-03-03 14:09:05.192299 7fd3b686c940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 -2> 2016-03-03 14:09:05.198884 7fd3b686c940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -1> 2016-03-03 14:09:05.200572 7fd3b686c940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 0> 2016-03-03 14:09:05.202443 7fd3b686c940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7fd3b686c940 time 2016-03-03 14:09:05.200585 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fd3b6362f2b] 2: (()+0x2c2f80) [0x7fd3b5d68f80] 3: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a] 5: (FileStore::mount()+0x3b42) [0x7fd3b602fa62] 6: (OSD::init()+0x26d) [0x7fd3b5de8a5d] 7: (main()+0x2954) [0x7fd3b5d6e474] 8: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5] 9: (()+0x2f82b7) [0x7fd3b5d9e2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:05.204762 7fd3b686c940 -1 *** Caught signal (Aborted) ** in thread 7fd3b686c940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7fd3b62762ca] 2: (()+0x10340) [0x7fd3b498a340] 3: (gsignal()+0x39) [0x7fd3b2c05cc9] 4: (abort()+0x148) [0x7fd3b2c090d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd3b3510535] 6: (()+0x5e6d6) [0x7fd3b350e6d6] 7: (()+0x5d799) [0x7fd3b350d799] 8: (__gxx_personality_v0()+0x52a) [0x7fd3b350e34a] 9: (()+0xfff3) [0x7fd3b2fa3ff3] 10: (_Unwind_Resume()+0x57) [0x7fd3b2fa4517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7fd3b6363136] 12: (()+0x2c2f80) [0x7fd3b5d68f80] 13: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a] 15: (FileStore::mount()+0x3b42) [0x7fd3b602fa62] 16: (OSD::init()+0x26d) [0x7fd3b5de8a5d] 17: (main()+0x2954) [0x7fd3b5d6e474] 18: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5] 19: (()+0x2f82b7) [0x7fd3b5d9e2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2016-03-03 14:09:05.204762 7fd3b686c940 -1 *** Caught signal (Aborted) ** in thread 7fd3b686c940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7fd3b62762ca] 2: (()+0x10340) [0x7fd3b498a340] 3: (gsignal()+0x39) [0x7fd3b2c05cc9] 4: (abort()+0x148) [0x7fd3b2c090d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd3b3510535] 6: (()+0x5e6d6) [0x7fd3b350e6d6] 7: (()+0x5d799) [0x7fd3b350d799] 8: (__gxx_personality_v0()+0x52a) [0x7fd3b350e34a] 9: (()+0xfff3) [0x7fd3b2fa3ff3] 10: (_Unwind_Resume()+0x57) [0x7fd3b2fa4517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7fd3b6363136] 12: (()+0x2c2f80) [0x7fd3b5d68f80] 13: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a] 15: (FileStore::mount()+0x3b42) [0x7fd3b602fa62] 16: (OSD::init()+0x26d) [0x7fd3b5de8a5d] 17: (main()+0x2954) [0x7fd3b5d6e474] 18: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5] 19: (()+0x2f82b7) [0x7fd3b5d9e2b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:05.392819 7f31118d0940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5807 2016-03-03 14:09:05.401988 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) 2016-03-03 14:09:05.402339 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2016-03-03 14:09:05.402345 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2016-03-03 14:09:05.402360 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported 2016-03-03 14:09:05.413273 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2016-03-03 14:09:05.413332 7f31118d0940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 2016-03-03 14:09:05.421311 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 2016-03-03 14:09:05.424378 7f31118d0940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f31118d0940 time 2016-03-03 14:09:05.422528 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f31113c6f2b] 2: (()+0x2c2f80) [0x7f3110dccf80] 3: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a] 5: (FileStore::mount()+0x3b42) [0x7f3111093a62] 6: (OSD::init()+0x26d) [0x7f3110e4ca5d] 7: (main()+0x2954) [0x7f3110dd2474] 8: (__libc_start_main()+0xf5) [0x7f310dc54ec5] 9: (()+0x2f82b7) [0x7f3110e022b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -24> 2016-03-03 14:09:05.390559 7f31118d0940 5 asok(0x7f31148b8000) register_command perfcounters_dump hook 0x7f3114838050 -23> 2016-03-03 14:09:05.390587 7f31118d0940 5 asok(0x7f31148b8000) register_command 1 hook 0x7f3114838050 -22> 2016-03-03 14:09:05.390589 7f31118d0940 5 asok(0x7f31148b8000) register_command perf dump hook 0x7f3114838050 -21> 2016-03-03 14:09:05.390606 7f31118d0940 5 asok(0x7f31148b8000) register_command perfcounters_schema hook 0x7f3114838050 -20> 2016-03-03 14:09:05.390607 7f31118d0940 5 asok(0x7f31148b8000) register_command 2 hook 0x7f3114838050 -19> 2016-03-03 14:09:05.390609 7f31118d0940 5 asok(0x7f31148b8000) register_command perf schema hook 0x7f3114838050 -18> 2016-03-03 14:09:05.390610 7f31118d0940 5 asok(0x7f31148b8000) register_command perf reset hook 0x7f3114838050 -17> 2016-03-03 14:09:05.390613 7f31118d0940 5 asok(0x7f31148b8000) register_command config show hook 0x7f3114838050 -16> 2016-03-03 14:09:05.390621 7f31118d0940 5 asok(0x7f31148b8000) register_command config set hook 0x7f3114838050 -15> 2016-03-03 14:09:05.390623 7f31118d0940 5 asok(0x7f31148b8000) register_command config get hook 0x7f3114838050 -14> 2016-03-03 14:09:05.390624 7f31118d0940 5 asok(0x7f31148b8000) register_command config diff hook 0x7f3114838050 -13> 2016-03-03 14:09:05.390625 7f31118d0940 5 asok(0x7f31148b8000) register_command log flush hook 0x7f3114838050 -12> 2016-03-03 14:09:05.390632 7f31118d0940 5 asok(0x7f31148b8000) register_command log dump hook 0x7f3114838050 -11> 2016-03-03 14:09:05.390634 7f31118d0940 5 asok(0x7f31148b8000) register_command log reopen hook 0x7f3114838050 -10> 2016-03-03 14:09:05.392755 7f31118d0940 0 set uid:gid to 64045:64045 -9> 2016-03-03 14:09:05.392819 7f31118d0940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5807 -8> 2016-03-03 14:09:05.401988 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342) -7> 2016-03-03 14:09:05.402339 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -6> 2016-03-03 14:09:05.402345 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option -5> 2016-03-03 14:09:05.402360 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported -4> 2016-03-03 14:09:05.413273 7f31118d0940 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -3> 2016-03-03 14:09:05.413332 7f31118d0940 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5 -2> 2016-03-03 14:09:05.421311 7f31118d0940 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -1> 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal 0> 2016-03-03 14:09:05.424378 7f31118d0940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f31118d0940 time 2016-03-03 14:09:05.422528 os/FileJournal.h: 406: FAILED assert(fd == -1) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f31113c6f2b] 2: (()+0x2c2f80) [0x7f3110dccf80] 3: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e] 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a] 5: (FileStore::mount()+0x3b42) [0x7f3111093a62] 6: (OSD::init()+0x26d) [0x7f3110e4ca5d] 7: (main()+0x2954) [0x7f3110dd2474] 8: (__libc_start_main()+0xf5) [0x7f310dc54ec5] 9: (()+0x2f82b7) [0x7f3110e022b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events --- 2016-03-03 14:09:05.426790 7f31118d0940 -1 *** Caught signal (Aborted) ** in thread 7f31118d0940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7f31112da2ca] 2: (()+0x10340) [0x7f310f9ee340] 3: (gsignal()+0x39) [0x7f310dc69cc9] 4: (abort()+0x148) [0x7f310dc6d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f310e574535] 6: (()+0x5e6d6) [0x7f310e5726d6] 7: (()+0x5d799) [0x7f310e571799] 8: (__gxx_personality_v0()+0x52a) [0x7f310e57234a] 9: (()+0xfff3) [0x7f310e007ff3] 10: (_Unwind_Resume()+0x57) [0x7f310e008517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7f31113c7136] 12: (()+0x2c2f80) [0x7f3110dccf80] 13: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a] 15: (FileStore::mount()+0x3b42) [0x7f3111093a62] 16: (OSD::init()+0x26d) [0x7f3110e4ca5d] 17: (main()+0x2954) [0x7f3110dd2474] 18: (__libc_start_main()+0xf5) [0x7f310dc54ec5] 19: (()+0x2f82b7) [0x7f3110e022b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2016-03-03 14:09:05.426790 7f31118d0940 -1 *** Caught signal (Aborted) ** in thread 7f31118d0940 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7f31112da2ca] 2: (()+0x10340) [0x7f310f9ee340] 3: (gsignal()+0x39) [0x7f310dc69cc9] 4: (abort()+0x148) [0x7f310dc6d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f310e574535] 6: (()+0x5e6d6) [0x7f310e5726d6] 7: (()+0x5d799) [0x7f310e571799] 8: (__gxx_personality_v0()+0x52a) [0x7f310e57234a] 9: (()+0xfff3) [0x7f310e007ff3] 10: (_Unwind_Resume()+0x57) [0x7f310e008517] 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7f31113c7136] 12: (()+0x2c2f80) [0x7f3110dccf80] 13: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e] 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a] 15: (FileStore::mount()+0x3b42) [0x7f3111093a62] 16: (OSD::init()+0x26d) [0x7f3110e4ca5d] 17: (main()+0x2954) [0x7f3110dd2474] 18: (__libc_start_main()+0xf5) [0x7f310dc54ec5] 19: (()+0x2f82b7) [0x7f3110e022b7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 0 lockdep 0/ 0 context 0/ 0 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 0 buffer 0/ 0 timer 0/ 0 filer 0/ 1 striper 0/ 0 objecter 0/ 0 rados 0/ 0 rbd 0/ 5 rbd_replay 0/ 0 journaler 0/ 5 objectcacher 0/ 0 client 0/ 0 osd 0/ 0 optracker 0/ 0 objclass 0/ 0 filestore 1/ 3 keyvaluestore 0/ 0 journal 0/ 0 ms 0/ 0 mon 0/ 0 monc 0/ 0 paxos 0/ 0 tp 0/ 0 auth 1/ 5 crypto 0/ 0 finisher 0/ 0 heartbeatmap 0/ 0 perfcounter 0/ 0 rgw 1/10 civetweb 1/ 5 javaclient 0/ 0 asok 0/ 0 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.2.log --- end dump of recent events ---
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com