|
Hi All, I cannot start the OSD daemon, and need your helps. Any advice is appreciated. When I deployed the ceph cluster through ceph-deploy, it worked. But after some days, all the OSD daemons were down, and I could not start them. Then I redeployed it some times, and it was still this case. The environment: OS: CentOS 6.6 Ceph: hammer(0.92.2) When the OSD daemon was down, I started the OSD daemon on the OSD node, and the console outputted: [root@node1 ~]# service ceph start === osd.0 === libust[2285/2285]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) create-or-move updated item name 'osd.0' weight 0.11 at location {host=node1,root=default} to crush map Starting Ceph osd.0 on node1... libust[2329/2329]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal [root@node1 ~]# And the log file of ceph, /var/log/ceph/ceph-osd.0.log, recorded: 2015-09-07 15:13:02.433546 7f74cea0d800 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045 2015-09-07 15:13:02.508300 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53) 2015-09-07 15:13:02.511270 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work 2015-09-07 15:13:02.511293 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-09-07 15:13:02.520826 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported 2015-09-07 15:13:02.523890 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs 2015-09-07 15:13:02.527387 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-09-07 15:13:02.528890 7f74cea0d800 1 2015-09-07 15:13:02.528890 7f74cea0d800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530273 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530487 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory 2015-09-07 15:13:02.530770 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0) 2015-09-07 15:13:02.530798 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir 2015-09-07 15:13:02.530801 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "3.5_head", "oid": "5\/\/head\/\/3" }, { "op_num": 1, "op_name": "rmcoll", "collection": "3.5_head" } ] } "op_name": "remove", "collection": "3.5_head", "oid": "5\/\/head\/\/3" }, { "op_num": 1, "op_name": "rmcoll", "collection": "3.5_head" } ] } 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848 os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794] 3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e] 4: (FileStore::mount()+0x2d0e) [0x963d5e] 5: (OSD::init()+0x273) [0x6891f3] 6: (main()+0x384f) [0x62e10f] 7: (__libc_start_main()+0xfd) [0x3c70c1ed5d] 8: /usr/bin/ceph-osd() [0x629819] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -126> 2015-09-07 15:13:02.431001 7f74cea0d800 5 asok(0x5630000) register_command perfcounters_dump hook 0x55a8050 -125> 2015-09-07 15:13:02.431054 7f74cea0d800 5 asok(0x5630000) register_command 1 hook 0x55a8050 -126> 2015-09-07 15:13:02.431001 7f74cea0d800 5 asok(0x5630000) register_command perfcounters_dump hook 0x55a8050 -125> 2015-09-07 15:13:02.431054 7f74cea0d800 5 asok(0x5630000) register_command 1 hook 0x55a8050 -124> 2015-09-07 15:13:02.431059 7f74cea0d800 5 asok(0x5630000) register_command perf dump hook 0x55a8050 -124> 2015-09-07 15:13:02.431059 7f74cea0d800 5 asok(0x5630000) register_command perf dump hook 0x55a8050 -123> 2015-09-07 15:13:02.431070 7f74cea0d800 5 asok(0x5630000) register_command perfcounters_schema hook 0x55a8050 -122> 2015-09-07 15:13:02.431075 7f74cea0d800 5 -122> 2015-09-07 15:13:02.431075 7f74cea0d800 5 asok(0x5630000) register_command 2 hook 0x55a8050 -121> 2015-09-07 15:13:02.431085 7f74cea0d800 5 asok(0x5630000) register_command perf schema hook 0x55a8050 -121> 2015-09-07 15:13:02.431085 7f74cea0d800 5 asok(0x5630000) register_command perf schema hook 0x55a8050 -120> 2015-09-07 15:13:02.431089 7f74cea0d800 5 asok(0x5630000) register_command perf reset hook 0x55a8050 -119> 2015-09-07 15:13:02.431097 7f74cea0d800 5 asok(0x5630000) register_command config show hook 0x55a8050 -119> 2015-09-07 15:13:02.431097 7f74cea0d800 5 asok(0x5630000) register_command config show hook 0x55a8050 -118> 2015-09-07 15:13:02.431101 7f74cea0d800 5 asok(0x5630000) register_command config set hook 0x55a8050 -118> 2015-09-07 15:13:02.431101 7f74cea0d800 5 asok(0x5630000) register_command config set hook 0x55a8050 -117> 2015-09-07 15:13:02.431109 7f74cea0d800 5 asok(0x5630000) register_command config get hook 0x55a8050 -116> 2015-09-07 15:13:02.431112 7f74cea0d800 5 asok(0x5630000) register_command config diff hook 0x55a8050 -115> 2015-09-07 15:13:02.431121 7f74cea0d800 5 asok(0x5630000) register_command log flush hook 0x55a8050 -117> 2015-09-07 15:13:02.431109 7f74cea0d800 5 asok(0x5630000) register_command config get hook 0x55a8050 -116> 2015-09-07 15:13:02.431112 7f74cea0d800 5 asok(0x5630000) register_command config diff hook 0x55a8050 -115> 2015-09-07 15:13:02.431121 7f74cea0d800 5 asok(0x5630000) register_command log flush hook 0x55a8050 -114> 2015-09-07 15:13:02.431124 7f74cea0d800 5 asok(0x5630000) register_command log dump hook 0x55a8050 asok(0x5630000) register_command log dump hook 0x55a8050 -113> 2015-09-07 15:13:02.431133 7f74cea0d800 5 asok(0x5630000) register_command log reopen hook 0x55a8050 asok(0x5630000) register_command log reopen hook 0x55a8050 -112> 2015-09-07 15:13:02.433546 7f74cea0d800 0 -112> 2015-09-07 15:13:02.433546 7f74cea0d800 0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045 -111> 2015-09-07 15:13:02.434302 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/1045 need_addr=1accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/1045 need_addr=1 -110> 2015-09-07 15:13:02.434355 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/1045 need_addr=1 -109> 2015-09-07 15:13:02.434380 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/1045 need_addr=1accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/1045 need_addr=1 -108> 2015-09-07 15:13:02.434403 7f74cea0d800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1 -107> 2015-09-07 15:13:02.435349 7f74cea0d800 1 finished global_init_daemonize -107> 2015-09-07 15:13:02.435349 7f74cea0d800 1 finished global_init_daemonize -106> 2015-09-07 15:13:02.491120 7f74cea0d800 5 asok(0x5630000) init /var/run/ceph/ceph-osd.0.asok -106> 2015-09-07 15:13:02.491120 7f74cea0d800 5 asok(0x5630000) init /var/run/ceph/ceph-osd.0.asok -105> 2015-09-07 15:13:02.491164 7f74cea0d800 5 asok(0x5630000) bind_and_listen /var/run/ceph/ceph-osd.0.asok -105> 2015-09-07 15:13:02.491164 7f74cea0d800 5 asok(0x5630000) bind_and_listen /var/run/ceph/ceph-osd.0.asok -104> 2015-09-07 15:13:02.491404 7f74cea0d800 5 asok(0x5630000) register_command 0 hook 0x55a00b0 -104> 2015-09-07 15:13:02.491404 7f74cea0d800 5 asok(0x5630000) register_command 0 hook 0x55a00b0 -103> 2015-09-07 15:13:02.491429 7f74cea0d800 5 asok(0x5630000) register_command version hook 0x55a00b0 -103> 2015-09-07 15:13:02.491429 7f74cea0d800 5 asok(0x5630000) register_command version hook 0x55a00b0 -102> 2015-09-07 15:13:02.491435 7f74cea0d800 5 asok(0x5630000) register_command git_version hook 0x55a00b0 -102> 2015-09-07 15:13:02.491435 7f74cea0d800 5 asok(0x5630000) register_command git_version hook 0x55a00b0 -101> 2015-09-07 15:13:02.491441 7f74cea0d800 5 f74cea0d800 5 asok(0x5630000) register_command help hook 0x55a8150 -100> 2015-09-07 15:13:02.491445 7f74cea0d800 5 asok(0x5630000) register_command get_command_descriptions hook 0x55a8140 -99> 2015-09-07 15:13:02.491545 7f74cea0d800 10 monclient(hunting): build_initial_monmap -98> 2015-09-07 15:13:02.491563 7f74cd141700 5 asok(0x5630000) entry start -97> 2015-09-07 15:13:02.506754 7f74cea0d800 5 adding auth protocol: cephx -96> 2015-09-07 15:13:02.506788 7f74cea0d800 5 adding auth protocol: cephx -95> 2015-09-07 15:13:02.507056 7f74cea0d800 5 asok(0x5630000) register_command objecter_requests hook 0x55a8180 -94> 2015-09-07 15:13:02.507286 7f74cea0d800 1 -- 0.0.0.0:6800/1045 messenger.start -93> 2015-09-07 15:13:02.507367 7f74cea0d800 1 -- :/0 messenger.start -92> 2015-09-07 15:13:02.507504 7f74cea0d800 1 -- 0.0.0.0:6803/1045 messenger.start -91> 2015-09-07 15:13:02.507637 7f74cea0d800 1 -- 0.0.0.0:6802/1045 messenger.start -90> 2015-09-07 15:13:02.507744 7f74cea0d800 1 -- 0.0.0.0:6801/1045 messenger.start -89> 2015-09-07 15:13:02.507889 7f74cea0d800 1 -- :/0 messenger.start -88> 2015-09-07 15:13:02.508178 7f74cea0d800 2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal -87> 2015-09-07 15:13:02.508300 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53) -86> 2015-09-07 15:13:02.511270 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work -85> 2015-09-07 15:13:02.511293 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work -85> 2015-09-07 15:13:02.511293 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -84> 2015-09-07 15:13:02.520826 7f74cea0d800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported -83> 2015-09-07 15:13:02.523890 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrsfilestore(/var/lib/ceph/osd/ceph-0) limited size xattrs -82> 2015-09-07 15:13:02.527387 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled -81> 2015-09-07 15:13:02.528819 7f74cea0d800 2 -81> 2015-09-07 15:13:02.528819 7f74cea0d800 2 journal open /var/lib/ceph/osd/ceph-0/journal fsid c5f917c4-fd48-4bbf-954a-ab1ee1b524ad fs_op_seq 4411 -80> 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway -79> 2015-09-07 15:13:02.528890 7f74cea0d800 1 -79> 2015-09-07 15:13:02.528890 7f74cea0d800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 -78> 2015-09-07 15:13:02.528954 7f74cea0d800 2 journal read_entry 85811200 : seq 4412 3703 bytes journal read_entry 85811200 : seq 4412 3703 bytes -77> 2015-09-07 15:13:02.528966 7f74cea0d800 2 journal read_entry 85811200 : seq 4412 3703 bytes -77> 2015-09-07 15:13:02.528966 7f74cea0d800 2 journal read_entry 85811200 : seq 4412 3703 bytes -76> 2015-09-07 15:13:02.528971 7f74cea0d800 3 -76> 2015-09-07 15:13:02.528971 7f74cea0d800 3 journal journal_replay: applying op seq 4412 -75> 2015-09-07 15:13:02.529399 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4412journal journal_replay: r = 0, op_seq now 4412 -74> 2015-09-07 15:13:02.529432 7f74cea0d800 2 journal read_entry 85815296 : seq 4413 959 bytes journal read_entry 85815296 : seq 4413 959 bytes -73> 2015-09-07 15:13:02.529436 7f74cea0d800 3 journal journal_replay: applying op seq 4413 -73> 2015-09-07 15:13:02.529436 7f74cea0d800 3 journal journal_replay: applying op seq 4413 -72> 2015-09-07 15:13:02.529476 7f74cea0d800 3 -72> 2015-09-07 15:13:02.529476 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4413 -71> 2015-09-07 15:13:02.529501 7f74cea0d800 2 journal read_entry 85819392 : seq 4414 46 bytes -70> 2015-09-07 15:13:02.529573 7f74cea0d800 3 -70> 2015-09-07 15:13:02.529573 7f74cea0d800 3 journal journal_replay: applying op seq 4414 -69> 2015-09-07 15:13:02.529579 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4414 -69> 2015-09-07 15:13:02.529579 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4414 -68> 2015-09-07 15:13:02.529590 7f74cea0d800 2 journal read_entry 85823488 : seq 4415 959 bytes journal read_entry 85823488 : seq 4415 959 bytes -67> 2015-09-07 15:13:02.529592 7f74cea0d800 3 -67> 2015-09-07 15:13:02.529592 7f74cea0d800 3 journal journal_replay: applying op seq 4415 -66> 2015-09-07 15:13:02.529620 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4415 -65> 2015-09-07 15:13:02.529645 7f74cea0d800 2 -65> 2015-09-07 15:13:02.529645 7f74cea0d800 2 journal read_entry 85827584 : seq 4416 959 bytes -64> 2015-09-07 15:13:02.529648 7f74cea0d800 3 journal journal_replay: applying op -64> 2015-09-07 15:13:02.529648 7f74cea0d800 3 journal journal_replay: applying op seq 4416 seq 4416 -63> 2015-09-07 15:13:02.529672 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4416 -62> 2015-09-07 15:13:02.529687 7f74cea0d800 2 journal read_entry 85831680 : seq 4417 959 bytes -61> 2015-09-07 15:13:02.529690 7f74cea0d800 3 journal journal_replay: applying op seq 4417 -61> 2015-09-07 15:13:02.529690 7f74cea0d800 3 journal journal_replay: applying op seq 4417 -60> 2015-09-07 15:13:02.529711 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4417 journal journal_replay: r = 0, op_seq now 4417 -59> 2015-09-07 15:13:02.529776 7f74cea0d800 2 -59> 2015-09-07 15:13:02.529776 7f74cea0d800 2 journal read_entry 85835776 : seq 4418 959 bytes -58> 2015-09-07 15:13:02.529779 7f74cea0d800 3 journal journal_replay: applying op seq 4418 journal journal_replay: applying op seq 4418 -57> 2015-09-07 15:13:02.529803 7f74cea0d800 3 -57> 2015-09-07 15:13:02.529803 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4418 -56> 2015-09-07 15:13:02.529819 7f74cea0d800 2 journal read_entry 85839872 : seq 4419 959 bytesjournal read_entry 85839872 : seq 4419 959 bytes -55> 2015-09-07 15:13:02.529822 7f74cea0d800 3 journal journal_replay: applying op seq 4419 journal journal_replay: applying op seq 4419 -54> 2015-09-07 15:13:02.529842 7f74cea0d800 3 -54> 2015-09-07 15:13:02.529842 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4419 -53> 2015-09-07 15:13:02.529857 7f74cea0d800 2 journal read_entry 85843968 : seq 4420 259 bytes -52> 2015-09-07 15:13:02.529860 7f74cea0d800 3 journal journal_replay: applying op seq 4420 -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory -50> 2015-09-07 15:13:02.529946 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4420 -49> 2015-09-07 15:13:02.529958 7f74cea0d800 2 journal read_entry 85848064 : seq 4421 46 bytes -48> 2015-09-07 15:13:02.529962 7f74cea0d800 3 journal journal_replay: applying op seq 4421 -47> 2015-09-07 15:13:02.529967 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4421 -46> 2015-09-07 15:13:02.529976 7f74cea0d800 2 journal read_entry 85852160 : seq 4422 259 bytes -45> 2015-09-07 15:13:02.529979 7f74cea0d800 3 journal journal_replay: applying op seq 4422 -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory -43> 2015-09-07 15:13:02.530070 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4422 -42> 2015-09-07 15:13:02.530082 7f74cea0d800 2 journal read_entry 85856256 : seq 4423 46 bytes -41> 2015-09-07 15:13:02.530085 7f74cea0d800 3 journal journal_replay: applying op seq 4423 -40> 2015-09-07 15:13:02.530088 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4423 -39> 2015-09-07 15:13:02.530097 7f74cea0d800 2 -53> 2015-09-07 15:13:02.529857 7f74cea0d800 2 journal read_entry 85843968 : seq 4420 259 bytes -52> 2015-09-07 15:13:02.529860 7f74cea0d800 3 journal journal_replay: applying op seq 4420 -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory -50> 2015-09-07 15:13:02.529946 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4420 -49> 2015-09-07 15:13:02.529958 7f74cea0d800 2 journal read_entry 85848064 : seq 4421 46 bytes -48> 2015-09-07 15:13:02.529962 7f74cea0d800 3 journal journal_replay: applying op seq 4421 -47> 2015-09-07 15:13:02.529967 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4421 -46> 2015-09-07 15:13:02.529976 7f74cea0d800 2 journal read_entry 85852160 : seq 4422 259 bytes -45> 2015-09-07 15:13:02.529979 7f74cea0d800 3 journal journal_replay: applying op seq 4422 -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory -43> 2015-09-07 15:13:02.530070 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4422 -42> 2015-09-07 15:13:02.530082 7f74cea0d800 2 journal read_entry 85856256 : seq 4423 46 bytes -41> 2015-09-07 15:13:02.530085 7f74cea0d800 3 journal journal_replay: applying op seq 4423 -40> 2015-09-07 15:13:02.530088 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4423 -39> 2015-09-07 15:13:02.530097 7f74cea0d800 2 journal read_entry 85860352 : seq 4424 959 bytes -38> 2015-09-07 15:13:02.530100 7f74cea0d800 3 journal journal_replay: applying op seq 4424 -37> 2015-09-07 15:13:02.530135 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4424 -38> 2015-09-07 15:13:02.530100 7f74cea0d800 3 journal journal_replay: applying op seq 4424 -37> 2015-09-07 15:13:02.530135 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4424 -36> 2015-09-07 15:13:02.530152 7f74cea0d800 2 journal read_entry 85864448 : seq 4425 959 bytes -36> 2015-09-07 15:13:02.530152 7f74cea0d800 2 journal read_entry 85864448 : seq 4425 959 bytes -35> 2015-09-07 15:13:02.530155 7f74cea0d800 3 -35> 2015-09-07 15:13:02.530155 7f74cea0d800 3 journal journal_replay: applying op seq 4425 -34> 2015-09-07 15:13:02.530232 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4425journal journal_replay: r = 0, op_seq now 4425 -33> 2015-09-07 15:13:02.530245 7f74cea0d800 2 journal read_entry 85868544 : seq 4426 259 bytes journal read_entry 85868544 : seq 4426 259 bytes -32> 2015-09-07 15:13:02.530251 7f74cea0d800 3 -32> 2015-09-07 15:13:02.530251 7f74cea0d800 3 journal journal_replay: applying op seq 4426 -31> 2015-09-07 15:13:02.530273 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directoryfilestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directory -30> 2015-09-07 15:13:02.530299 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4426 journal journal_replay: r = 0, op_seq now 4426 -29> 2015-09-07 15:13:02.530310 7f74cea0d800 2 -29> 2015-09-07 15:13:02.530310 7f74cea0d800 2 journal read_entry 85872640 : seq 4427 46 bytes -28> 2015-09-07 15:13:02.530312 7f74cea0d800 3 journal journal_replay: applying op seq 4427 -27> 2015-09-07 15:13:02.530316 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4427 -27> 2015-09-07 15:13:02.530316 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4427 -26> 2015-09-07 15:13:02.530325 7f74cea0d800 2 journal read_entry 85876736 : seq 4428 259 bytes -26> 2015-09-07 15:13:02.530325 7f74cea0d800 2 journal read_entry 85876736 : seq 4428 259 bytes -25> 2015-09-07 15:13:02.530328 7f74cea0d800 3 journal journal_replay: applying op seq 4428 -25> 2015-09-07 15:13:02.530328 7f74cea0d800 3 journal journal_replay: applying op seq 4428 -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1 -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2) No such file or directory 0//head//3 in index: (2) No such file or directory -23> 2015-09-07 15:13:02.530389 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4428 -22> 2015-09-07 15:13:02.530401 7f74cea0d800 2 journal read_entry 85880832 : seq 4429 46 bytes -21> 2015-09-07 15:13:02.530404 7f74cea0d800 3 -21> 2015-09-07 15:13:02.530404 7f74cea0d800 3 journal journal_replay: applying op seq 4429 -20> 2015-09-07 15:13:02.530448 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4429journal journal_replay: r = 0, op_seq now 4429 -19> 2015-09-07 15:13:02.530458 7f74cea0d800 2 journal read_entry 85884928 : seq 4430 259 bytes journal read_entry 85884928 : seq 4430 259 bytes -18> 2015-09-07 15:13:02.530465 7f74cea0d800 3 journal journal_replay: applying op seq 4430 -18> 2015-09-07 15:13:02.530465 7f74cea0d800 3 journal journal_replay: applying op seq 4430 -17> 2015-09-07 15:13:02.530487 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory -16> 2015-09-07 15:13:02.530514 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4430 -15> 2015-09-07 15:13:02.530525 7f74cea0d800 2 journal read_entry 85889024 : seq 4431 46 bytes -14> 2015-09-07 15:13:02.530528 7f74cea0d800 3 journal journal_replay: applying op seq 4431 -13> 2015-09-07 15:13:02.530532 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4431 -12> 2015-09-07 15:13:02.530541 7f74cea0d800 2 journal read_entry 85893120 : seq 4432 259 bytes -11> 2015-09-07 15:13:02.530549 7f74cea0d800 3 journal journal_replay: applying op seq 4432 -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory -9> 2015-09-07 15:13:02.530593 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4432 -8> 2015-09-07 15:13:02.530604 7f74cea0d800 2 journal read_entry 85897216 : seq 4433 46 bytes -7> 2015-09-07 15:13:02.530606 7f74cea0d800 3 journal journal_replay: applying op seq 4433 -6> 2015-09-07 15:13:02.530610 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4433 -5> 2015-09-07 15:13:02.530621 7f74cea0d800 2 journal read_entry 85901312 : seq 4434 259 bytes -4> 2015-09-07 15:13:02.530624 7f74cea0d800 3 journal journal_replay: applying op seq 4434 -3> 2015-09-07 15:13:02.530770 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0) -2> 2015-09-07 15:13:02.530798 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir -1> 2015-09-07 15:13:02.530801 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "3.5_head", "oid": "5\/\/head\/\/3" }, { "op_num": 1, "op_name": "rmcoll", "collection": "3.5_head" } ] } 0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848 os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794] 3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e] 4: (FileStore::mount()+0x2d0e) [0x963d5e] 5: (OSD::init()+0x273) [0x6891f3] 6: (main()+0x384f) [0x62e10f] 7: (__libc_start_main()+0xfd) [0x3c70c1ed5d] 8: /usr/bin/ceph-osd() [0x629819] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd -16> 2015-09-07 15:13:02.530514 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4430 -15> 2015-09-07 15:13:02.530525 7f74cea0d800 2 journal read_entry 85889024 : seq 4431 46 bytes -14> 2015-09-07 15:13:02.530528 7f74cea0d800 3 journal journal_replay: applying op seq 4431 -13> 2015-09-07 15:13:02.530532 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4431 -12> 2015-09-07 15:13:02.530541 7f74cea0d800 2 journal read_entry 85893120 : seq 4432 259 bytes -11> 2015-09-07 15:13:02.530549 7f74cea0d800 3 journal journal_replay: applying op seq 4432 -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory -9> 2015-09-07 15:13:02.530593 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4432 -8> 2015-09-07 15:13:02.530604 7f74cea0d800 2 journal read_entry 85897216 : seq 4433 46 bytes -7> 2015-09-07 15:13:02.530606 7f74cea0d800 3 journal journal_replay: applying op seq 4433 -6> 2015-09-07 15:13:02.530610 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now 4433 -5> 2015-09-07 15:13:02.530621 7f74cea0d800 2 journal read_entry 85901312 : seq 4434 259 bytes -4> 2015-09-07 15:13:02.530624 7f74cea0d800 3 journal journal_replay: applying op seq 4434 -3> 2015-09-07 15:13:02.530770 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0) -2> 2015-09-07 15:13:02.530798 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir -1> 2015-09-07 15:13:02.530801 7f74cea0d800 0 filestore(/var/lib/ceph/osd/ceph-0) transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "3.5_head", "oid": "5\/\/head\/\/3" }, { "op_num": 1, "op_name": "rmcoll", "collection": "3.5_head" } ] } 0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848 os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error") ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06] 2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794] 3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e] 4: (FileStore::mount()+0x2d0e) [0x963d5e] 5: (OSD::init()+0x273) [0x6891f3] 6: (main()+0x384f) [0x62e10f] 7: (__libc_start_main()+0xfd) [0x3c70c1ed5d] 8: /usr/bin/ceph-osd() [0x629819] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- 2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) ** in thread 7f74cea0d800 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: /usr/bin/ceph-osd() [0xa476a5] 2: /lib64/libpthread.so.0() [0x3c7100f710] 3: (gsignal()+0x35) [0x3c70c32625] 4: (abort()+0x175) [0x3c70c33e05] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) ** in thread 7f74cea0d800 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: /usr/bin/ceph-osd() [0xa476a5] 2: /lib64/libpthread.so.0() [0x3c7100f710] 3: (gsignal()+0x35) [0x3c70c32625] 4: (abort()+0x175) [0x3c70c33e05] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.0.log --- end dump of recent events --- I hope you could help me. Any advice is appreciated. Thanks, Aaron |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
