Rooney,
Just tried hooking up osd.0 back. osd.0 seems to be better as I was able to run ceph-objectstore-tool export so decided to try hooking it up. Looks like journal is not happy. Is there any way to get this running? Or do I need to start getting data using ceph-objectstore-tool?
2017-09-13 18:51:50.051421 7f44dd847800 0 set uid:gid to 1001:1001 (ceph:ceph)
2017-09-13 18:51:50.051435 7f44dd847800 0 ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0), process ceph-osd, pid 3899
2017-09-13 18:51:50.052323 7f44dd847800 0 pidfile_write: ignore empty --pid-file
2017-09-13 18:51:50.061586 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
2017-09-13 18:51:50.061823 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2017-09-13 18:51:50.061826 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
2017-09-13 18:51:50.061838 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice is supported
2017-09-13 18:51:50.077506 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2017-09-13 18:51:50.077549 7f44dd847800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
2017-09-13 18:51:50.078066 7f44dd847800 1 leveldb: Recovering log #28069
2017-09-13 18:51:50.177610 7f44dd847800 1 leveldb: Delete type=0 #28069
2017-09-13 18:51:50.177708 7f44dd847800 1 leveldb: Delete type=3 #28068
2017-09-13 18:51:57.946233 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2017-09-13 18:51:57.947293 7f44dd847800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 18: 9998729216 bytes, block size 4096 bytes, directio = 1, aio = 1
2017-09-13 18:51:57.949835 7f44dd847800 -1 journal Unable to read past sequence 27057121 but header indicates the journal has committed up through 27057593, journal is corrupt
2017-09-13 18:51:57.951824 7f44dd847800 -1 os/filestore/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f44dd847800 time 2017-09-13 18:51:57.949837
os/filestore/FileJournal.cc: 2036: FAILED assert(0)
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x82) [0x55c809640d02]
2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4]
3: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95]
4: (FileStore::mount()+0x2e28) [0x55c8092d0a88]
5: (OSD::init()+0x27d) [0x55c808f697ed]
6: (main()+0x2a64) [0x55c808ed05d4]
7: (__libc_start_main()+0xf5) [0x7f44da6e3b45]
8: (()+0x341117) [0x55c808f1b117]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-59> 2017-09-13 18:51:50.043283 7f44dd847800 5 asok(0x55c813d76000) register_command perfcounters_dump hook 0x55c813cbe030
-58> 2017-09-13 18:51:50.043312 7f44dd847800 5 asok(0x55c813d76000) register_command 1 hook 0x55c813cbe030
-57> 2017-09-13 18:51:50.043317 7f44dd847800 5 asok(0x55c813d76000) register_command perf dump hook 0x55c813cbe030
-56> 2017-09-13 18:51:50.043322 7f44dd847800 5 asok(0x55c813d76000) register_command perfcounters_schema hook 0x55c813cbe030
-55> 2017-09-13 18:51:50.043326 7f44dd847800 5 asok(0x55c813d76000) register_command 2 hook 0x55c813cbe030
-54> 2017-09-13 18:51:50.043330 7f44dd847800 5 asok(0x55c813d76000) register_command perf schema hook 0x55c813cbe030
-53> 2017-09-13 18:51:50.043334 7f44dd847800 5 asok(0x55c813d76000) register_command perf reset hook 0x55c813cbe030
-52> 2017-09-13 18:51:50.043339 7f44dd847800 5 asok(0x55c813d76000) register_command config show hook 0x55c813cbe030
-51> 2017-09-13 18:51:50.043344 7f44dd847800 5 asok(0x55c813d76000) register_command config set hook 0x55c813cbe030
-50> 2017-09-13 18:51:50.043349 7f44dd847800 5 asok(0x55c813d76000) register_command config get hook 0x55c813cbe030
-49> 2017-09-13 18:51:50.043355 7f44dd847800 5 asok(0x55c813d76000) register_command config diff hook 0x55c813cbe030
-48> 2017-09-13 18:51:50.043361 7f44dd847800 5 asok(0x55c813d76000) register_command log flush hook 0x55c813cbe030
-47> 2017-09-13 18:51:50.043367 7f44dd847800 5 asok(0x55c813d76000) register_command log dump hook 0x55c813cbe030
-46> 2017-09-13 18:51:50.043373 7f44dd847800 5 asok(0x55c813d76000) register_command log reopen hook 0x55c813cbe030
-45> 2017-09-13 18:51:50.051421 7f44dd847800 0 set uid:gid to 1001:1001 (ceph:ceph)
-44> 2017-09-13 18:51:50.051435 7f44dd847800 0 ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0), process ceph-osd, pid 3899
-43> 2017-09-13 18:51:50.051455 7f44dd847800 5 object store type is filestore
-42> 2017-09-13 18:51:50.052283 7f44dd847800 1 -- 192.168.1.30:0/0 learned my addr 192.168.1.30:0/0
-41> 2017-09-13 18:51:50.052290 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.1.30:6800/3899 need_addr=0
-40> 2017-09-13 18:51:50.052297 7f44dd847800 1 -- 192.168.2.30:0/0 learned my addr 192.168.2.30:0/0
-39> 2017-09-13 18:51:50.052300 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.2.30:6800/3899 need_addr=0
-38> 2017-09-13 18:51:50.052309 7f44dd847800 1 -- 192.168.2.30:0/0 learned my addr 192.168.2.30:0/0
-37> 2017-09-13 18:51:50.052313 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.2.30:6801/3899 need_addr=0
-36> 2017-09-13 18:51:50.052318 7f44dd847800 1 -- 192.168.1.30:0/0 learned my addr 192.168.1.30:0/0
-35> 2017-09-13 18:51:50.052321 7f44dd847800 1 accepter.accepter.bind my_inst.addr is 192.168.1.30:6801/3899 need_addr=0
-34> 2017-09-13 18:51:50.052323 7f44dd847800 0 pidfile_write: ignore empty --pid-file
-33> 2017-09-13 18:51:50.054120 7f44dd847800 5 asok(0x55c813d76000) init /var/run/ceph/ceph-osd.0.asok
-32> 2017-09-13 18:51:50.054131 7f44dd847800 5 asok(0x55c813d76000) bind_and_listen /var/run/ceph/ceph-osd.0.asok
-31> 2017-09-13 18:51:50.054205 7f44dd847800 5 asok(0x55c813d76000) register_command 0 hook 0x55c813cba0d0
-30> 2017-09-13 18:51:50.054211 7f44dd847800 5 asok(0x55c813d76000) register_command version hook 0x55c813cba0d0
-29> 2017-09-13 18:51:50.054215 7f44dd847800 5 asok(0x55c813d76000) register_command git_version hook 0x55c813cba0d0
-28> 2017-09-13 18:51:50.054219 7f44dd847800 5 asok(0x55c813d76000) register_command help hook 0x55c813cbe1e0
-27> 2017-09-13 18:51:50.054223 7f44dd847800 5 asok(0x55c813d76000) register_command get_command_descriptions hook 0x55c813cbe1f0
-26> 2017-09-13 18:51:50.054257 7f44d6fff700 5 asok(0x55c813d76000) entry start
-25> 2017-09-13 18:51:50.054288 7f44dd847800 10 monclient(hunting): build_initial_monmap
-24> 2017-09-13 18:51:50.061212 7f44dd847800 5 adding auth protocol: none
-23> 2017-09-13 18:51:50.061222 7f44dd847800 5 adding auth protocol: none
-22> 2017-09-13 18:51:50.061376 7f44dd847800 5 asok(0x55c813d76000) register_command objecter_requests hook 0x55c813cbe2b0
-21> 2017-09-13 18:51:50.061427 7f44dd847800 1 -- 192.168.1.30:6800/3899 messenger.start
-20> 2017-09-13 18:51:50.061446 7f44dd847800 1 -- :/0 messenger.start
-19> 2017-09-13 18:51:50.061460 7f44dd847800 1 -- 192.168.1.30:6801/3899 messenger.start
-18> 2017-09-13 18:51:50.061472 7f44dd847800 1 -- 192.168.2.30:6801/3899 messenger.start
-17> 2017-09-13 18:51:50.061484 7f44dd847800 1 -- 192.168.2.30:6800/3899 messenger.start
-16> 2017-09-13 18:51:50.061497 7f44dd847800 1 -- :/0 messenger.start
-15> 2017-09-13 18:51:50.061540 7f44dd847800 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
-14> 2017-09-13 18:51:50.061586 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342)
-13> 2017-09-13 18:51:50.061823 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-12> 2017-09-13 18:51:50.061826 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
-11> 2017-09-13 18:51:50.061838 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice is supported
-10> 2017-09-13 18:51:50.077506 7f44dd847800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-9> 2017-09-13 18:51:50.077549 7f44dd847800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf
-8> 2017-09-13 18:51:50.078066 7f44dd847800 1 leveldb: Recovering log #28069
-7> 2017-09-13 18:51:50.177610 7f44dd847800 1 leveldb: Delete type=0 #28069
-6> 2017-09-13 18:51:50.177708 7f44dd847800 1 leveldb: Delete type=3 #28068
-5> 2017-09-13 18:51:57.946233 7f44dd847800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-4> 2017-09-13 18:51:57.947259 7f44dd847800 2 journal open /var/lib/ceph/osd/ceph-0/journal fsid f626317c-6e12-49d3-98ab-23d799430d30 fs_op_seq 27057120
-3> 2017-09-13 18:51:57.947293 7f44dd847800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 18: 9998729216 bytes, block size 4096 bytes, directio = 1, aio = 1
-2> 2017-09-13 18:51:57.949828 7f44dd847800 2 journal No further valid entries found, journal is most likely valid
-1> 2017-09-13 18:51:57.949835 7f44dd847800 -1 journal Unable to read past sequence 27057121 but header indicates the journal has committed up through 27057593, journal is corrupt
0> 2017-09-13 18:51:57.951824 7f44dd847800 -1 os/filestore/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f44dd847800 time 2017-09-13 18:51:57.949837
os/filestore/FileJournal.cc: 2036: FAILED assert(0)
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x82) [0x55c809640d02]
2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4]
3: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95]
4: (FileStore::mount()+0x2e28) [0x55c8092d0a88]
5: (OSD::init()+0x27d) [0x55c808f697ed]
6: (main()+0x2a64) [0x55c808ed05d4]
7: (__libc_start_main()+0xf5) [0x7f44da6e3b45]
8: (()+0x341117) [0x55c808f1b117]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2017-09-13 18:51:57.954585 7f44dd847800 -1 *** Caught signal (Aborted) **
in thread 7f44dd847800 thread_name:ceph-osd
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
1: (()+0x960e57) [0x55c80953ae57]
2: (()+0xf8d0) [0x7f44dc6b58d0]
3: (gsignal()+0x37) [0x7f44da6f7067]
4: (abort()+0x148) [0x7f44da6f8448]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x55c809640ed6]
6: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4]
7: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95]
8: (FileStore::mount()+0x2e28) [0x55c8092d0a88]
9: (OSD::init()+0x27d) [0x55c808f697ed]
10: (main()+0x2a64) [0x55c808ed05d4]
11: (__libc_start_main()+0xf5) [0x7f44da6e3b45]
12: (()+0x341117) [0x55c808f1b117]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2017-09-13 18:51:57.954585 7f44dd847800 -1 *** Caught signal (Aborted) **
in thread 7f44dd847800 thread_name:ceph-osd
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
1: (()+0x960e57) [0x55c80953ae57]
2: (()+0xf8d0) [0x7f44dc6b58d0]
3: (gsignal()+0x37) [0x7f44da6f7067]
4: (abort()+0x148) [0x7f44da6f8448]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x55c809640ed6]
6: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xa84) [0x55c8093c4da4]
7: (JournalingObjectStore::journal_replay(unsigned long)+0x205) [0x55c8092feb95]
8: (FileStore::mount()+0x2e28) [0x55c8092d0a88]
9: (OSD::init()+0x27d) [0x55c808f697ed]
10: (main()+0x2a64) [0x55c808ed05d4]
11: (__libc_start_main()+0xf5) [0x7f44da6e3b45]
12: (()+0x341117) [0x55c808f1b117]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 newstore
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
1/ 5 kinetic
1/ 5 fuse
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
Regards,
Hong
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
