OSDs go down with infernalis

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I'm (almost) a new user of ceph (couple of month). In my university, we start to
do some test with ceph a couple of months ago.

We have 2 clusters. Each cluster have 100 OSDs on 10 servers :

Each server as this setup :

CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Memory : 128GB of Memory
OS Storage : 2 x SSD 240GB Intel S3500 DC (raid 1)
Journal Storage : 2 x SSD 400GB Intel S3300 DC (no Raid)
OSD Disk : 10 x HGST ultrastar-7k6000 6TB
Network : 1 x 10Gb/s
OS : Ubuntu 14.04
Ceph version : infernalis 9.2.0

One cluster give access to some user through a S3 gateway (service is still in
beta). We call this cluster "ceph-beta".

One cluster is for our internal need to learn more about ceph. We call this
cluster "ceph-test". (those servers will be integrated into the ceph-beta
cluster when we will need more space)

We have deploy both clusters with the ceph-ansible playbook[1]

Journal are raw partitions on SSDs (400GB Intel S3300 DC) with no raid. 5
journals partitions on each SSDs.

OSDs disk are format in XFS.

1. https://github.com/ceph/ceph-ansible

We have an issue. Some OSDs go down and don't start. It seem to be related to
the fsid of the journal partition :

> -1> 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal

in attachment, the full logs of one of the dead OSDs

We had this issue with 2 OSDs on ceph-beta cluster fixed by removing, zapping
and readding it.

Now, we have the same issue on ceph-test cluster but on 18 OSDs.

Now the stats of this cluster

> root@icadmin004:~# ceph -s
>     cluster 4fb4773c-0873-44ad-a65f-269f01bfcff8
>      health HEALTH_WARN
>             1024 pgs incomplete
>             1024 pgs stuck inactive
>             1024 pgs stuck unclean
>      monmap e1: 3 mons at {iccluster003=10.90.37.4:6789/0,iccluster014=10.90.37.15:6789/0,iccluster022=10.90.37.23:6789/0}
>             election epoch 62, quorum 0,1,2 iccluster003,iccluster014,iccluster022
>      osdmap e242: 100 osds: 82 up, 82 in
>             flags sortbitwise
>       pgmap v469212: 2304 pgs, 10 pools, 2206 bytes data, 181 objects
>             4812 MB used, 447 TB / 447 TB avail
>                 1280 active+clean
>                 1024 creating+incomplete

We have install this cluster at the begin of February. We did not use that
cluster at all even at the begin to troubleshoot an issue with ceph-ansible. We
did not push any data neither create pool. What could explain this behaviour ?

Thanks for your help

Best regards,

-- 
Yoann Moulin
EPFL IC-IT
2016-03-03 14:09:00.433074 7efd1a9d5940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4446
2016-03-03 14:09:01.315583 7efd1a9d5940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:01.338328 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:01.338335 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:01.338362 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:01.341468 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:01.341517 7efd1a9d5940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:01.411145 7efd1a9d5940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:01.692400 7efd1a9d5940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:01.694251 7efd1a9d5940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7efd1a9d5940 time 2016-03-03 14:09:01.692413
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7efd1a4cbf2b]
 2: (()+0x2c2f80) [0x7efd19ed1f80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
 5: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
 6: (OSD::init()+0x26d) [0x7efd19f51a5d]
 7: (main()+0x2954) [0x7efd19ed7474]
 8: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
 9: (()+0x2f82b7) [0x7efd19f072b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -24> 2016-03-03 14:09:00.430918 7efd1a9d5940  5 asok(0x7efd1e460000) register_command perfcounters_dump hook 0x7efd1e3e0050
   -23> 2016-03-03 14:09:00.430937 7efd1a9d5940  5 asok(0x7efd1e460000) register_command 1 hook 0x7efd1e3e0050
   -22> 2016-03-03 14:09:00.430938 7efd1a9d5940  5 asok(0x7efd1e460000) register_command perf dump hook 0x7efd1e3e0050
   -21> 2016-03-03 14:09:00.430940 7efd1a9d5940  5 asok(0x7efd1e460000) register_command perfcounters_schema hook 0x7efd1e3e0050
   -20> 2016-03-03 14:09:00.430941 7efd1a9d5940  5 asok(0x7efd1e460000) register_command 2 hook 0x7efd1e3e0050
   -19> 2016-03-03 14:09:00.430943 7efd1a9d5940  5 asok(0x7efd1e460000) register_command perf schema hook 0x7efd1e3e0050
   -18> 2016-03-03 14:09:00.430945 7efd1a9d5940  5 asok(0x7efd1e460000) register_command perf reset hook 0x7efd1e3e0050
   -17> 2016-03-03 14:09:00.430947 7efd1a9d5940  5 asok(0x7efd1e460000) register_command config show hook 0x7efd1e3e0050
   -16> 2016-03-03 14:09:00.430950 7efd1a9d5940  5 asok(0x7efd1e460000) register_command config set hook 0x7efd1e3e0050
   -15> 2016-03-03 14:09:00.430954 7efd1a9d5940  5 asok(0x7efd1e460000) register_command config get hook 0x7efd1e3e0050
   -14> 2016-03-03 14:09:00.430955 7efd1a9d5940  5 asok(0x7efd1e460000) register_command config diff hook 0x7efd1e3e0050
   -13> 2016-03-03 14:09:00.430957 7efd1a9d5940  5 asok(0x7efd1e460000) register_command log flush hook 0x7efd1e3e0050
   -12> 2016-03-03 14:09:00.430958 7efd1a9d5940  5 asok(0x7efd1e460000) register_command log dump hook 0x7efd1e3e0050
   -11> 2016-03-03 14:09:00.430959 7efd1a9d5940  5 asok(0x7efd1e460000) register_command log reopen hook 0x7efd1e3e0050
   -10> 2016-03-03 14:09:00.433016 7efd1a9d5940  0 set uid:gid to 64045:64045
    -9> 2016-03-03 14:09:00.433074 7efd1a9d5940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4446
    -8> 2016-03-03 14:09:01.315583 7efd1a9d5940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
    -7> 2016-03-03 14:09:01.338328 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
    -6> 2016-03-03 14:09:01.338335 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
    -5> 2016-03-03 14:09:01.338362 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
    -4> 2016-03-03 14:09:01.341468 7efd1a9d5940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
    -3> 2016-03-03 14:09:01.341517 7efd1a9d5940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
    -2> 2016-03-03 14:09:01.411145 7efd1a9d5940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
    -1> 2016-03-03 14:09:01.692400 7efd1a9d5940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
     0> 2016-03-03 14:09:01.694251 7efd1a9d5940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7efd1a9d5940 time 2016-03-03 14:09:01.692413
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7efd1a4cbf2b]
 2: (()+0x2c2f80) [0x7efd19ed1f80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
 5: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
 6: (OSD::init()+0x26d) [0x7efd19f51a5d]
 7: (main()+0x2954) [0x7efd19ed7474]
 8: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
 9: (()+0x2f82b7) [0x7efd19f072b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:01.696632 7efd1a9d5940 -1 *** Caught signal (Aborted) **
 in thread 7efd1a9d5940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7efd1a3df2ca]
 2: (()+0x10340) [0x7efd18af3340]
 3: (gsignal()+0x39) [0x7efd16d6ecc9]
 4: (abort()+0x148) [0x7efd16d720d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd17679535]
 6: (()+0x5e6d6) [0x7efd176776d6]
 7: (()+0x5d799) [0x7efd17676799]
 8: (__gxx_personality_v0()+0x52a) [0x7efd1767734a]
 9: (()+0xfff3) [0x7efd1710cff3]
 10: (_Unwind_Resume()+0x57) [0x7efd1710d517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7efd1a4cc136]
 12: (()+0x2c2f80) [0x7efd19ed1f80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
 15: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
 16: (OSD::init()+0x26d) [0x7efd19f51a5d]
 17: (main()+0x2954) [0x7efd19ed7474]
 18: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
 19: (()+0x2f82b7) [0x7efd19f072b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-03-03 14:09:01.696632 7efd1a9d5940 -1 *** Caught signal (Aborted) **
 in thread 7efd1a9d5940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7efd1a3df2ca]
 2: (()+0x10340) [0x7efd18af3340]
 3: (gsignal()+0x39) [0x7efd16d6ecc9]
 4: (abort()+0x148) [0x7efd16d720d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd17679535]
 6: (()+0x5e6d6) [0x7efd176776d6]
 7: (()+0x5d799) [0x7efd17676799]
 8: (__gxx_personality_v0()+0x52a) [0x7efd1767734a]
 9: (()+0xfff3) [0x7efd1710cff3]
 10: (_Unwind_Resume()+0x57) [0x7efd1710d517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7efd1a4cc136]
 12: (()+0x2c2f80) [0x7efd19ed1f80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7efd1a1b476e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7efd1a1c353a]
 15: (FileStore::mount()+0x3b42) [0x7efd1a198a62]
 16: (OSD::init()+0x26d) [0x7efd19f51a5d]
 17: (main()+0x2954) [0x7efd19ed7474]
 18: (__libc_start_main()+0xf5) [0x7efd16d59ec5]
 19: (()+0x2f82b7) [0x7efd19f072b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:01.889329 7ff0a521d940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4750
2016-03-03 14:09:01.897263 7ff0a521d940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:01.897561 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:01.897565 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:01.897576 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:01.918907 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:01.918937 7ff0a521d940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:01.965269 7ff0a521d940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:01.966401 7ff0a521d940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:01.968379 7ff0a521d940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ff0a521d940 time 2016-03-03 14:09:01.966412
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff0a4d13f2b]
 2: (()+0x2c2f80) [0x7ff0a4719f80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
 5: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
 6: (OSD::init()+0x26d) [0x7ff0a4799a5d]
 7: (main()+0x2954) [0x7ff0a471f474]
 8: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
 9: (()+0x2f82b7) [0x7ff0a474f2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -24> 2016-03-03 14:09:01.887108 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command perfcounters_dump hook 0x7ff0a7caa050
   -23> 2016-03-03 14:09:01.887136 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command 1 hook 0x7ff0a7caa050
   -22> 2016-03-03 14:09:01.887139 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command perf dump hook 0x7ff0a7caa050
   -21> 2016-03-03 14:09:01.887141 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command perfcounters_schema hook 0x7ff0a7caa050
   -20> 2016-03-03 14:09:01.887142 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command 2 hook 0x7ff0a7caa050
   -19> 2016-03-03 14:09:01.887144 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command perf schema hook 0x7ff0a7caa050
   -18> 2016-03-03 14:09:01.887147 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command perf reset hook 0x7ff0a7caa050
   -17> 2016-03-03 14:09:01.887148 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command config show hook 0x7ff0a7caa050
   -16> 2016-03-03 14:09:01.887150 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command config set hook 0x7ff0a7caa050
   -15> 2016-03-03 14:09:01.887153 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command config get hook 0x7ff0a7caa050
   -14> 2016-03-03 14:09:01.887154 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command config diff hook 0x7ff0a7caa050
   -13> 2016-03-03 14:09:01.887159 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command log flush hook 0x7ff0a7caa050
   -12> 2016-03-03 14:09:01.887160 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command log dump hook 0x7ff0a7caa050
   -11> 2016-03-03 14:09:01.887161 7ff0a521d940  5 asok(0x7ff0a7d2a000) register_command log reopen hook 0x7ff0a7caa050
   -10> 2016-03-03 14:09:01.889275 7ff0a521d940  0 set uid:gid to 64045:64045
    -9> 2016-03-03 14:09:01.889329 7ff0a521d940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 4750
    -8> 2016-03-03 14:09:01.897263 7ff0a521d940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
    -7> 2016-03-03 14:09:01.897561 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
    -6> 2016-03-03 14:09:01.897565 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
    -5> 2016-03-03 14:09:01.897576 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
    -4> 2016-03-03 14:09:01.918907 7ff0a521d940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
    -3> 2016-03-03 14:09:01.918937 7ff0a521d940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
    -2> 2016-03-03 14:09:01.965269 7ff0a521d940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
    -1> 2016-03-03 14:09:01.966401 7ff0a521d940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
     0> 2016-03-03 14:09:01.968379 7ff0a521d940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7ff0a521d940 time 2016-03-03 14:09:01.966412
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff0a4d13f2b]
 2: (()+0x2c2f80) [0x7ff0a4719f80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
 5: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
 6: (OSD::init()+0x26d) [0x7ff0a4799a5d]
 7: (main()+0x2954) [0x7ff0a471f474]
 8: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
 9: (()+0x2f82b7) [0x7ff0a474f2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:01.970874 7ff0a521d940 -1 *** Caught signal (Aborted) **
 in thread 7ff0a521d940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7ff0a4c272ca]
 2: (()+0x10340) [0x7ff0a333b340]
 3: (gsignal()+0x39) [0x7ff0a15b6cc9]
 4: (abort()+0x148) [0x7ff0a15ba0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a1ec1535]
 6: (()+0x5e6d6) [0x7ff0a1ebf6d6]
 7: (()+0x5d799) [0x7ff0a1ebe799]
 8: (__gxx_personality_v0()+0x52a) [0x7ff0a1ebf34a]
 9: (()+0xfff3) [0x7ff0a1954ff3]
 10: (_Unwind_Resume()+0x57) [0x7ff0a1955517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7ff0a4d14136]
 12: (()+0x2c2f80) [0x7ff0a4719f80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
 15: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
 16: (OSD::init()+0x26d) [0x7ff0a4799a5d]
 17: (main()+0x2954) [0x7ff0a471f474]
 18: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
 19: (()+0x2f82b7) [0x7ff0a474f2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-03-03 14:09:01.970874 7ff0a521d940 -1 *** Caught signal (Aborted) **
 in thread 7ff0a521d940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7ff0a4c272ca]
 2: (()+0x10340) [0x7ff0a333b340]
 3: (gsignal()+0x39) [0x7ff0a15b6cc9]
 4: (abort()+0x148) [0x7ff0a15ba0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a1ec1535]
 6: (()+0x5e6d6) [0x7ff0a1ebf6d6]
 7: (()+0x5d799) [0x7ff0a1ebe799]
 8: (__gxx_personality_v0()+0x52a) [0x7ff0a1ebf34a]
 9: (()+0xfff3) [0x7ff0a1954ff3]
 10: (_Unwind_Resume()+0x57) [0x7ff0a1955517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7ff0a4d14136]
 12: (()+0x2c2f80) [0x7ff0a4719f80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7ff0a49fc76e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7ff0a4a0b53a]
 15: (FileStore::mount()+0x3b42) [0x7ff0a49e0a62]
 16: (OSD::init()+0x26d) [0x7ff0a4799a5d]
 17: (main()+0x2954) [0x7ff0a471f474]
 18: (__libc_start_main()+0xf5) [0x7ff0a15a1ec5]
 19: (()+0x2f82b7) [0x7ff0a474f2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.160375 7fd3b686c940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5719
2016-03-03 14:09:05.169748 7fd3b686c940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:05.170085 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:05.170091 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:05.170107 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:05.192222 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:05.192299 7fd3b686c940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:05.198884 7fd3b686c940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:05.200572 7fd3b686c940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:05.202443 7fd3b686c940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7fd3b686c940 time 2016-03-03 14:09:05.200585
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fd3b6362f2b]
 2: (()+0x2c2f80) [0x7fd3b5d68f80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
 5: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
 6: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
 7: (main()+0x2954) [0x7fd3b5d6e474]
 8: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
 9: (()+0x2f82b7) [0x7fd3b5d9e2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -24> 2016-03-03 14:09:05.158253 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command perfcounters_dump hook 0x7fd3ba926050
   -23> 2016-03-03 14:09:05.158272 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command 1 hook 0x7fd3ba926050
   -22> 2016-03-03 14:09:05.158274 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command perf dump hook 0x7fd3ba926050
   -21> 2016-03-03 14:09:05.158276 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command perfcounters_schema hook 0x7fd3ba926050
   -20> 2016-03-03 14:09:05.158277 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command 2 hook 0x7fd3ba926050
   -19> 2016-03-03 14:09:05.158279 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command perf schema hook 0x7fd3ba926050
   -18> 2016-03-03 14:09:05.158280 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command perf reset hook 0x7fd3ba926050
   -17> 2016-03-03 14:09:05.158282 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command config show hook 0x7fd3ba926050
   -16> 2016-03-03 14:09:05.158283 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command config set hook 0x7fd3ba926050
   -15> 2016-03-03 14:09:05.158285 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command config get hook 0x7fd3ba926050
   -14> 2016-03-03 14:09:05.158286 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command config diff hook 0x7fd3ba926050
   -13> 2016-03-03 14:09:05.158287 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command log flush hook 0x7fd3ba926050
   -12> 2016-03-03 14:09:05.158288 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command log dump hook 0x7fd3ba926050
   -11> 2016-03-03 14:09:05.158290 7fd3b686c940  5 asok(0x7fd3ba9a6000) register_command log reopen hook 0x7fd3ba926050
   -10> 2016-03-03 14:09:05.160308 7fd3b686c940  0 set uid:gid to 64045:64045
    -9> 2016-03-03 14:09:05.160375 7fd3b686c940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5719
    -8> 2016-03-03 14:09:05.169748 7fd3b686c940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
    -7> 2016-03-03 14:09:05.170085 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
    -6> 2016-03-03 14:09:05.170091 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
    -5> 2016-03-03 14:09:05.170107 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
    -4> 2016-03-03 14:09:05.192222 7fd3b686c940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
    -3> 2016-03-03 14:09:05.192299 7fd3b686c940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
    -2> 2016-03-03 14:09:05.198884 7fd3b686c940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
    -1> 2016-03-03 14:09:05.200572 7fd3b686c940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
     0> 2016-03-03 14:09:05.202443 7fd3b686c940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7fd3b686c940 time 2016-03-03 14:09:05.200585
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7fd3b6362f2b]
 2: (()+0x2c2f80) [0x7fd3b5d68f80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
 5: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
 6: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
 7: (main()+0x2954) [0x7fd3b5d6e474]
 8: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
 9: (()+0x2f82b7) [0x7fd3b5d9e2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.204762 7fd3b686c940 -1 *** Caught signal (Aborted) **
 in thread 7fd3b686c940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7fd3b62762ca]
 2: (()+0x10340) [0x7fd3b498a340]
 3: (gsignal()+0x39) [0x7fd3b2c05cc9]
 4: (abort()+0x148) [0x7fd3b2c090d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd3b3510535]
 6: (()+0x5e6d6) [0x7fd3b350e6d6]
 7: (()+0x5d799) [0x7fd3b350d799]
 8: (__gxx_personality_v0()+0x52a) [0x7fd3b350e34a]
 9: (()+0xfff3) [0x7fd3b2fa3ff3]
 10: (_Unwind_Resume()+0x57) [0x7fd3b2fa4517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7fd3b6363136]
 12: (()+0x2c2f80) [0x7fd3b5d68f80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
 15: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
 16: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
 17: (main()+0x2954) [0x7fd3b5d6e474]
 18: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
 19: (()+0x2f82b7) [0x7fd3b5d9e2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-03-03 14:09:05.204762 7fd3b686c940 -1 *** Caught signal (Aborted) **
 in thread 7fd3b686c940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7fd3b62762ca]
 2: (()+0x10340) [0x7fd3b498a340]
 3: (gsignal()+0x39) [0x7fd3b2c05cc9]
 4: (abort()+0x148) [0x7fd3b2c090d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd3b3510535]
 6: (()+0x5e6d6) [0x7fd3b350e6d6]
 7: (()+0x5d799) [0x7fd3b350d799]
 8: (__gxx_personality_v0()+0x52a) [0x7fd3b350e34a]
 9: (()+0xfff3) [0x7fd3b2fa3ff3]
 10: (_Unwind_Resume()+0x57) [0x7fd3b2fa4517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7fd3b6363136]
 12: (()+0x2c2f80) [0x7fd3b5d68f80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7fd3b604b76e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7fd3b605a53a]
 15: (FileStore::mount()+0x3b42) [0x7fd3b602fa62]
 16: (OSD::init()+0x26d) [0x7fd3b5de8a5d]
 17: (main()+0x2954) [0x7fd3b5d6e474]
 18: (__libc_start_main()+0xf5) [0x7fd3b2bf0ec5]
 19: (()+0x2f82b7) [0x7fd3b5d9e2b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.392819 7f31118d0940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5807
2016-03-03 14:09:05.401988 7f31118d0940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2016-03-03 14:09:05.402339 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-03-03 14:09:05.402345 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2016-03-03 14:09:05.402360 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
2016-03-03 14:09:05.413273 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2016-03-03 14:09:05.413332 7f31118d0940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
2016-03-03 14:09:05.421311 7f31118d0940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
2016-03-03 14:09:05.424378 7f31118d0940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f31118d0940 time 2016-03-03 14:09:05.422528
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f31113c6f2b]
 2: (()+0x2c2f80) [0x7f3110dccf80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
 5: (FileStore::mount()+0x3b42) [0x7f3111093a62]
 6: (OSD::init()+0x26d) [0x7f3110e4ca5d]
 7: (main()+0x2954) [0x7f3110dd2474]
 8: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
 9: (()+0x2f82b7) [0x7f3110e022b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -24> 2016-03-03 14:09:05.390559 7f31118d0940  5 asok(0x7f31148b8000) register_command perfcounters_dump hook 0x7f3114838050
   -23> 2016-03-03 14:09:05.390587 7f31118d0940  5 asok(0x7f31148b8000) register_command 1 hook 0x7f3114838050
   -22> 2016-03-03 14:09:05.390589 7f31118d0940  5 asok(0x7f31148b8000) register_command perf dump hook 0x7f3114838050
   -21> 2016-03-03 14:09:05.390606 7f31118d0940  5 asok(0x7f31148b8000) register_command perfcounters_schema hook 0x7f3114838050
   -20> 2016-03-03 14:09:05.390607 7f31118d0940  5 asok(0x7f31148b8000) register_command 2 hook 0x7f3114838050
   -19> 2016-03-03 14:09:05.390609 7f31118d0940  5 asok(0x7f31148b8000) register_command perf schema hook 0x7f3114838050
   -18> 2016-03-03 14:09:05.390610 7f31118d0940  5 asok(0x7f31148b8000) register_command perf reset hook 0x7f3114838050
   -17> 2016-03-03 14:09:05.390613 7f31118d0940  5 asok(0x7f31148b8000) register_command config show hook 0x7f3114838050
   -16> 2016-03-03 14:09:05.390621 7f31118d0940  5 asok(0x7f31148b8000) register_command config set hook 0x7f3114838050
   -15> 2016-03-03 14:09:05.390623 7f31118d0940  5 asok(0x7f31148b8000) register_command config get hook 0x7f3114838050
   -14> 2016-03-03 14:09:05.390624 7f31118d0940  5 asok(0x7f31148b8000) register_command config diff hook 0x7f3114838050
   -13> 2016-03-03 14:09:05.390625 7f31118d0940  5 asok(0x7f31148b8000) register_command log flush hook 0x7f3114838050
   -12> 2016-03-03 14:09:05.390632 7f31118d0940  5 asok(0x7f31148b8000) register_command log dump hook 0x7f3114838050
   -11> 2016-03-03 14:09:05.390634 7f31118d0940  5 asok(0x7f31148b8000) register_command log reopen hook 0x7f3114838050
   -10> 2016-03-03 14:09:05.392755 7f31118d0940  0 set uid:gid to 64045:64045
    -9> 2016-03-03 14:09:05.392819 7f31118d0940  0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5807
    -8> 2016-03-03 14:09:05.401988 7f31118d0940  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
    -7> 2016-03-03 14:09:05.402339 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
    -6> 2016-03-03 14:09:05.402345 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
    -5> 2016-03-03 14:09:05.402360 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: splice is supported
    -4> 2016-03-03 14:09:05.413273 7f31118d0940  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
    -3> 2016-03-03 14:09:05.413332 7f31118d0940  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: extsize is supported and your kernel >= 3.5
    -2> 2016-03-03 14:09:05.421311 7f31118d0940  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
    -1> 2016-03-03 14:09:05.422515 7f31118d0940 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected eeadbce2-f096-4156-ba56-dfc634e59106, invalid (someone else's?) journal
     0> 2016-03-03 14:09:05.424378 7f31118d0940 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f31118d0940 time 2016-03-03 14:09:05.422528
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f31113c6f2b]
 2: (()+0x2c2f80) [0x7f3110dccf80]
 3: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
 5: (FileStore::mount()+0x3b42) [0x7f3111093a62]
 6: (OSD::init()+0x26d) [0x7f3110e4ca5d]
 7: (main()+0x2954) [0x7f3110dd2474]
 8: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
 9: (()+0x2f82b7) [0x7f3110e022b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
2016-03-03 14:09:05.426790 7f31118d0940 -1 *** Caught signal (Aborted) **
 in thread 7f31118d0940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7f31112da2ca]
 2: (()+0x10340) [0x7f310f9ee340]
 3: (gsignal()+0x39) [0x7f310dc69cc9]
 4: (abort()+0x148) [0x7f310dc6d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f310e574535]
 6: (()+0x5e6d6) [0x7f310e5726d6]
 7: (()+0x5d799) [0x7f310e571799]
 8: (__gxx_personality_v0()+0x52a) [0x7f310e57234a]
 9: (()+0xfff3) [0x7f310e007ff3]
 10: (_Unwind_Resume()+0x57) [0x7f310e008517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7f31113c7136]
 12: (()+0x2c2f80) [0x7f3110dccf80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
 15: (FileStore::mount()+0x3b42) [0x7f3111093a62]
 16: (OSD::init()+0x26d) [0x7f3110e4ca5d]
 17: (main()+0x2954) [0x7f3110dd2474]
 18: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
 19: (()+0x2f82b7) [0x7f3110e022b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2016-03-03 14:09:05.426790 7f31118d0940 -1 *** Caught signal (Aborted) **
 in thread 7f31118d0940

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (()+0x7d02ca) [0x7f31112da2ca]
 2: (()+0x10340) [0x7f310f9ee340]
 3: (gsignal()+0x39) [0x7f310dc69cc9]
 4: (abort()+0x148) [0x7f310dc6d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f310e574535]
 6: (()+0x5e6d6) [0x7f310e5726d6]
 7: (()+0x5d799) [0x7f310e571799]
 8: (__gxx_personality_v0()+0x52a) [0x7f310e57234a]
 9: (()+0xfff3) [0x7f310e007ff3]
 10: (_Unwind_Resume()+0x57) [0x7f310e008517]
 11: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x296) [0x7f31113c7136]
 12: (()+0x2c2f80) [0x7f3110dccf80]
 13: (FileJournal::~FileJournal()+0x67e) [0x7f31110af76e]
 14: (JournalingObjectStore::journal_replay(unsigned long)+0xbfa) [0x7f31110be53a]
 15: (FileStore::mount()+0x3b42) [0x7f3111093a62]
 16: (OSD::init()+0x26d) [0x7f3110e4ca5d]
 17: (main()+0x2954) [0x7f3110dd2474]
 18: (__libc_start_main()+0xf5) [0x7f310dc54ec5]
 19: (()+0x2f82b7) [0x7f3110e022b7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.2.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux