-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
I would check that the /var/lib/ceph/osd/ceph-0/ is mounted and has
the file structure for Ceph.
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Mon, Sep 7, 2015 at 2:16 AM, Aaron wrote:
> Hi All,
>
> I cannot start the OSD daemon, and need your helps. Any advice is
> appreciated.
>
> When I deployed the ceph cluster through ceph-deploy, it worked. But after
> some days, all the OSD daemons were down, and I could not start them. Then I
> redeployed it some times, and it was still this case.
>
> The environment:
> OS: CentOS 6.6
> Ceph: hammer(0.92.2)
>
> When the OSD daemon was down, I started the OSD daemon on the OSD node, and
> the console outputted:
> [root@node1 ~]# service ceph start
> === osd.0 ===
> libust[2285/2285]: Warning: HOME environment variable not set. Disabling
> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
> create-or-move updated item name 'osd.0' weight 0.11 at location
> {host=node1,root=default} to crush map
> Starting Ceph osd.0 on node1...
> libust[2329/2329]: Warning: HOME environment variable not set. Disabling
> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
> starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0
> /var/lib/ceph/osd/ceph-0/journal
> [root@node1 ~]#
>
> And the log file of ceph, /var/log/ceph/ceph-osd.0.log, recorded:
> 2015-09-07 15:13:02.433546 7f74cea0d800 0 ceph version 0.94.2
> (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045
> 2015-09-07 15:13:02.508300 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53)
> 2015-09-07 15:13:02.511270 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP
> ioctl is supported and appears to work
> 2015-09-07 15:13:02.511293 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP
> ioctl is disabled via 'filestore fiemap' config option
>
> 2015-09-07 15:13:02.520826 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features:
> syscall(SYS_syncfs, fd) fully supported
> 2015-09-07 15:13:02.523890 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs
> 2015-09-07 15:13:02.527387 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode:
> checkpoint is not enabled
> 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open:
> disabling aio for non-block journal. Use journal_force_aio to force use of
> aio anyway
> 2015-09-07 15:13:02.528890 7f74cea0d800 1 2015-09-07 15:13:02.528890
> 7f74cea0d800 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19:
> 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
>
> 2015-09-07 15:13:02.529881 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2)
> No such file or directory
> 2015-09-07 15:13:02.530039 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2)
> No such file or directory
> 2015-09-07 15:13:02.530273 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2)
> No such file or directory
> 2015-09-07 15:13:02.530349 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2)
> No such file or directory
>
> 2015-09-07 15:13:02.530487 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2)
> No such file or directory
> 2015-09-07 15:13:02.530569 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2)
> No such file or directory
> 2015-09-07 15:13:02.530770 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not
> handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0)
> 2015-09-07 15:13:02.530798 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd
> data dir
>
> 2015-09-07 15:13:02.530801 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
> {
> "ops": [
> {
> "op_num": 0,
> "op_name": "remove",
> "collection": "3.5_head",
> "oid": "5\/\/head\/\/3"
> },
> {
> "op_num": 1,
> "op_name": "rmcoll",
> "collection": "3.5_head"
> }
> ]
> }
>
> "op_name": "remove",
> "collection": "3.5_head",
> "oid": "5\/\/head\/\/3"
> },
> {
> "op_num": 1,
> "op_name": "rmcoll",
> "collection": "3.5_head"
> }
> ]
> }
>
> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function
> 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&,
> uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07
> 15:13:02.530848
> os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")
>
> ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
> 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
> int, ThreadPool::TPHandle*)+0xc16) [0x975a06]
> 2: (FileStore::_do_transactions(std::list std::allocator >&, unsigned long,
> ThreadPool::TPHandle*)+0x64) [0x97d794]
> 3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
> 4: (FileStore::mount()+0x2d0e) [0x963d5e]
> 5: (OSD::init()+0x273) [0x6891f3]
> 6: (main()+0x384f) [0x62e10f]
> 7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
> 8: /usr/bin/ceph-osd() [0x629819]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> --- begin dump of recent events ---
> -126> 2015-09-07 15:13:02.431001 7f74cea0d800 5 asok(0x5630000)
> register_command perfcounters_dump hook 0x55a8050
> -125> 2015-09-07 15:13:02.431054 7f74cea0d800 5 asok(0x5630000)
> register_command 1 hook 0x55a8050
> -126> 2015-09-07 15:13:02.431001 7f74cea0d800 5 asok(0x5630000)
> register_command perfcounters_dump hook 0x55a8050
> -125> 2015-09-07 15:13:02.431054 7f74cea0d800 5 asok(0x5630000)
> register_command 1 hook 0x55a8050
> -124> 2015-09-07 15:13:02.431059 7f74cea0d800 5 asok(0x5630000)
> register_command perf dump hook 0x55a8050 -124> 2015-09-07 15:13:02.431059
> 7f74cea0d800 5 asok(0x5630000) register_command perf dump hook 0x55a8050
> -123> 2015-09-07 15:13:02.431070 7f74cea0d800 5 asok(0x5630000)
> register_command perfcounters_schema hook 0x55a8050
>
> -122> 2015-09-07 15:13:02.431075 7f74cea0d800 5 -122> 2015-09-07
> 15:13:02.431075 7f74cea0d800 5 asok(0x5630000) register_command 2 hook
> 0x55a8050
>
> -121> 2015-09-07 15:13:02.431085 7f74cea0d800 5 asok(0x5630000)
> register_command perf schema hook 0x55a8050 -121> 2015-09-07
> 15:13:02.431085 7f74cea0d800 5 asok(0x5630000) register_command perf schema
> hook 0x55a8050
> -120> 2015-09-07 15:13:02.431089 7f74cea0d800 5 asok(0x5630000)
> register_command perf reset hook 0x55a8050
> -119> 2015-09-07 15:13:02.431097 7f74cea0d800 5 asok(0x5630000)
> register_command config show hook 0x55a8050
> -119> 2015-09-07 15:13:02.431097 7f74cea0d800 5 asok(0x5630000)
> register_command config show hook 0x55a8050
> -118> 2015-09-07 15:13:02.431101 7f74cea0d800 5 asok(0x5630000)
> register_command config set hook 0x55a8050
> -118> 2015-09-07 15:13:02.431101 7f74cea0d800 5 asok(0x5630000)
> register_command config set hook 0x55a8050
> -117> 2015-09-07 15:13:02.431109 7f74cea0d800 5 asok(0x5630000)
> register_command config get hook 0x55a8050
> -116> 2015-09-07 15:13:02.431112 7f74cea0d800 5 asok(0x5630000)
> register_command config diff hook 0x55a8050
> -115> 2015-09-07 15:13:02.431121 7f74cea0d800 5 asok(0x5630000)
> register_command log flush hook 0x55a8050
> -117> 2015-09-07 15:13:02.431109 7f74cea0d800 5 asok(0x5630000)
> register_command config get hook 0x55a8050
> -116> 2015-09-07 15:13:02.431112 7f74cea0d800 5 asok(0x5630000)
> register_command config diff hook 0x55a8050
> -115> 2015-09-07 15:13:02.431121 7f74cea0d800 5 asok(0x5630000)
> register_command log flush hook 0x55a8050
> -114> 2015-09-07 15:13:02.431124 7f74cea0d800 5 asok(0x5630000)
> register_command log dump hook 0x55a8050
> asok(0x5630000) register_command log dump hook 0x55a8050
> -113> 2015-09-07 15:13:02.431133 7f74cea0d800 5 asok(0x5630000)
> register_command log reopen hook 0x55a8050
> asok(0x5630000) register_command log reopen hook 0x55a8050
> -112> 2015-09-07 15:13:02.433546 7f74cea0d800 0 -112> 2015-09-07
> 15:13:02.433546 7f74cea0d800 0 ceph version 0.94.2
> (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045
> -111> 2015-09-07 15:13:02.434302 7f74cea0d800 1 accepter.accepter.bind
> my_inst.addr is 0.0.0.0:6800/1045 need_addr=1accepter.accepter.bind
> my_inst.addr is 0.0.0.0:6800/1045 need_addr=1
> -110> 2015-09-07 15:13:02.434355 7f74cea0d800 1 accepter.accepter.bind
> my_inst.addr is 0.0.0.0:6801/1045 need_addr=1
> -109> 2015-09-07 15:13:02.434380 7f74cea0d800 1 accepter.accepter.bind
> my_inst.addr is 0.0.0.0:6802/1045 need_addr=1accepter.accepter.bind
> my_inst.addr is 0.0.0.0:6802/1045 need_addr=1
> -108> 2015-09-07 15:13:02.434403 7f74cea0d800 1 accepter.accepter.bind
> my_inst.addr is 0.0.0.0:6803/1045 need_addr=1
> accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1
> -107> 2015-09-07 15:13:02.435349 7f74cea0d800 1 finished
> global_init_daemonize
> -107> 2015-09-07 15:13:02.435349 7f74cea0d800 1 finished
> global_init_daemonize
> -106> 2015-09-07 15:13:02.491120 7f74cea0d800 5 asok(0x5630000) init
> /var/run/ceph/ceph-osd.0.asok
> -106> 2015-09-07 15:13:02.491120 7f74cea0d800 5 asok(0x5630000) init
> /var/run/ceph/ceph-osd.0.asok
> -105> 2015-09-07 15:13:02.491164 7f74cea0d800 5 asok(0x5630000)
> bind_and_listen /var/run/ceph/ceph-osd.0.asok
> -105> 2015-09-07 15:13:02.491164 7f74cea0d800 5 asok(0x5630000)
> bind_and_listen /var/run/ceph/ceph-osd.0.asok
> -104> 2015-09-07 15:13:02.491404 7f74cea0d800 5 asok(0x5630000)
> register_command 0 hook 0x55a00b0
> -104> 2015-09-07 15:13:02.491404 7f74cea0d800 5 asok(0x5630000)
> register_command 0 hook 0x55a00b0
> -103> 2015-09-07 15:13:02.491429 7f74cea0d800 5 asok(0x5630000)
> register_command version hook 0x55a00b0
> -103> 2015-09-07 15:13:02.491429 7f74cea0d800 5 asok(0x5630000)
> register_command version hook 0x55a00b0
> -102> 2015-09-07 15:13:02.491435 7f74cea0d800 5 asok(0x5630000)
> register_command git_version hook 0x55a00b0 -102> 2015-09-07
> 15:13:02.491435 7f74cea0d800 5 asok(0x5630000) register_command git_version
> hook 0x55a00b0
> -101> 2015-09-07 15:13:02.491441 7f74cea0d800 5 f74cea0d800 5
> asok(0x5630000) register_command help hook 0x55a8150
> -100> 2015-09-07 15:13:02.491445 7f74cea0d800 5 asok(0x5630000)
> register_command get_command_descriptions hook 0x55a8140
> -99> 2015-09-07 15:13:02.491545 7f74cea0d800 10 monclient(hunting):
> build_initial_monmap
> -98> 2015-09-07 15:13:02.491563 7f74cd141700 5 asok(0x5630000) entry
> start
> -97> 2015-09-07 15:13:02.506754 7f74cea0d800 5 adding auth protocol:
> cephx
> -96> 2015-09-07 15:13:02.506788 7f74cea0d800 5 adding auth protocol:
> cephx
> -95> 2015-09-07 15:13:02.507056 7f74cea0d800 5 asok(0x5630000)
> register_command objecter_requests hook 0x55a8180
> -94> 2015-09-07 15:13:02.507286 7f74cea0d800 1 -- 0.0.0.0:6800/1045
> messenger.start
> -93> 2015-09-07 15:13:02.507367 7f74cea0d800 1 -- :/0 messenger.start
> -92> 2015-09-07 15:13:02.507504 7f74cea0d800 1 -- 0.0.0.0:6803/1045
> messenger.start
> -91> 2015-09-07 15:13:02.507637 7f74cea0d800 1 -- 0.0.0.0:6802/1045
> messenger.start
> -90> 2015-09-07 15:13:02.507744 7f74cea0d800 1 -- 0.0.0.0:6801/1045
> messenger.start
> -89> 2015-09-07 15:13:02.507889 7f74cea0d800 1 -- :/0 messenger.start
> -88> 2015-09-07 15:13:02.508178 7f74cea0d800 2 osd.0 0 mounting
> /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
> -87> 2015-09-07 15:13:02.508300 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53)
> -86> 2015-09-07 15:13:02.511270 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP
> ioctl is supported and appears to work
> -85> 2015-09-07 15:13:02.511293 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP
> ioctl is disabled via 'filestore fiemap' config option
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP
> ioctl is supported and appears to work
> -85> 2015-09-07 15:13:02.511293 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP
> ioctl is disabled via 'filestore fiemap' config option
> -84> 2015-09-07 15:13:02.520826 7f74cea0d800 0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features:
> syscall(SYS_syncfs, fd) fully supported
>
> -83> 2015-09-07 15:13:02.523890 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) limited size
> xattrsfilestore(/var/lib/ceph/osd/ceph-0) limited size xattrs
> -82> 2015-09-07 15:13:02.527387 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode:
> checkpoint is not enabled
> filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode:
> checkpoint is not enabled
> -81> 2015-09-07 15:13:02.528819 7f74cea0d800 2 -81> 2015-09-07
> 15:13:02.528819 7f74cea0d800 2 journal open
> /var/lib/ceph/osd/ceph-0/journal fsid c5f917c4-fd48-4bbf-954a-ab1ee1b524ad
> fs_op_seq 4411
> -80> 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal
> FileJournal::_open: disabling aio for non-block journal. Use
> journal_force_aio to force use of aio anyway
>
> -79> 2015-09-07 15:13:02.528890 7f74cea0d800 1 -79> 2015-09-07
> 15:13:02.528890 7f74cea0d800 1 journal _open
> /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096
> bytes, directio = 1, aio = 0
> -78> 2015-09-07 15:13:02.528954 7f74cea0d800 2 journal read_entry
> 85811200 : seq 4412 3703 bytes
> journal read_entry 85811200 : seq 4412 3703 bytes
> -77> 2015-09-07 15:13:02.528966 7f74cea0d800 2 journal read_entry
> 85811200 : seq 4412 3703 bytes
> -77> 2015-09-07 15:13:02.528966 7f74cea0d800 2 journal read_entry
> 85811200 : seq 4412 3703 bytes
> -76> 2015-09-07 15:13:02.528971 7f74cea0d800 3 -76> 2015-09-07
> 15:13:02.528971 7f74cea0d800 3 journal journal_replay: applying op seq 4412
> -75> 2015-09-07 15:13:02.529399 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4412journal journal_replay: r = 0, op_seq now 4412
> -74> 2015-09-07 15:13:02.529432 7f74cea0d800 2 journal read_entry
> 85815296 : seq 4413 959 bytes
> journal read_entry 85815296 : seq 4413 959 bytes
> -73> 2015-09-07 15:13:02.529436 7f74cea0d800 3 journal journal_replay:
> applying op seq 4413
> -73> 2015-09-07 15:13:02.529436 7f74cea0d800 3 journal journal_replay:
> applying op seq 4413
> -72> 2015-09-07 15:13:02.529476 7f74cea0d800 3 -72> 2015-09-07
> 15:13:02.529476 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now
> 4413
> -71> 2015-09-07 15:13:02.529501 7f74cea0d800 2 journal read_entry
> 85819392 : seq 4414 46 bytes
> -70> 2015-09-07 15:13:02.529573 7f74cea0d800 3 -70> 2015-09-07
> 15:13:02.529573 7f74cea0d800 3 journal journal_replay: applying op seq 4414
>
> -69> 2015-09-07 15:13:02.529579 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4414 -69> 2015-09-07 15:13:02.529579 7f74cea0d800 3
> journal journal_replay: r = 0, op_seq now 4414
> -68> 2015-09-07 15:13:02.529590 7f74cea0d800 2 journal read_entry
> 85823488 : seq 4415 959 bytes
> journal read_entry 85823488 : seq 4415 959 bytes
> -67> 2015-09-07 15:13:02.529592 7f74cea0d800 3 -67> 2015-09-07
> 15:13:02.529592 7f74cea0d800 3 journal journal_replay: applying op seq 4415
> -66> 2015-09-07 15:13:02.529620 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4415
> -65> 2015-09-07 15:13:02.529645 7f74cea0d800 2 -65> 2015-09-07
> 15:13:02.529645 7f74cea0d800 2 journal read_entry 85827584 : seq 4416 959
> bytes
>
> -64> 2015-09-07 15:13:02.529648 7f74cea0d800 3 journal journal_replay:
> applying op -64> 2015-09-07 15:13:02.529648 7f74cea0d800 3 journal
> journal_replay: applying op seq 4416 seq 4416
> -63> 2015-09-07 15:13:02.529672 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4416
> -62> 2015-09-07 15:13:02.529687 7f74cea0d800 2 journal read_entry
> 85831680 : seq 4417 959 bytes
> -61> 2015-09-07 15:13:02.529690 7f74cea0d800 3 journal journal_replay:
> applying op seq 4417 -61> 2015-09-07 15:13:02.529690 7f74cea0d800 3
> journal journal_replay: applying op seq 4417
> -60> 2015-09-07 15:13:02.529711 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4417
> journal journal_replay: r = 0, op_seq now 4417
> -59> 2015-09-07 15:13:02.529776 7f74cea0d800 2 -59> 2015-09-07
> 15:13:02.529776 7f74cea0d800 2 journal read_entry 85835776 : seq 4418 959
> bytes
> -58> 2015-09-07 15:13:02.529779 7f74cea0d800 3 journal journal_replay:
> applying op seq 4418
> journal journal_replay: applying op seq 4418
> -57> 2015-09-07 15:13:02.529803 7f74cea0d800 3 -57> 2015-09-07
> 15:13:02.529803 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now
> 4418
> -56> 2015-09-07 15:13:02.529819 7f74cea0d800 2 journal read_entry
> 85839872 : seq 4419 959 bytesjournal read_entry 85839872 : seq 4419 959
> bytes
> -55> 2015-09-07 15:13:02.529822 7f74cea0d800 3 journal journal_replay:
> applying op seq 4419
> journal journal_replay: applying op seq 4419
> -54> 2015-09-07 15:13:02.529842 7f74cea0d800 3 -54> 2015-09-07
> 15:13:02.529842 7f74cea0d800 3 journal journal_replay: r = 0, op_seq now
> 4419
> -53> 2015-09-07 15:13:02.529857 7f74cea0d800 2 journal read_entry
> 85843968 : seq 4420 259 bytes
> -52> 2015-09-07 15:13:02.529860 7f74cea0d800 3 journal journal_replay:
> applying op seq 4420
> -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2)
> No such file or directory
> -50> 2015-09-07 15:13:02.529946 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4420
> -49> 2015-09-07 15:13:02.529958 7f74cea0d800 2 journal read_entry
> 85848064 : seq 4421 46 bytes
> -48> 2015-09-07 15:13:02.529962 7f74cea0d800 3 journal journal_replay:
> applying op seq 4421
> -47> 2015-09-07 15:13:02.529967 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4421
> -46> 2015-09-07 15:13:02.529976 7f74cea0d800 2 journal read_entry
> 85852160 : seq 4422 259 bytes
> -45> 2015-09-07 15:13:02.529979 7f74cea0d800 3 journal journal_replay:
> applying op seq 4422
> -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2)
> No such file or directory
> -43> 2015-09-07 15:13:02.530070 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4422
> -42> 2015-09-07 15:13:02.530082 7f74cea0d800 2 journal read_entry
> 85856256 : seq 4423 46 bytes
> -41> 2015-09-07 15:13:02.530085 7f74cea0d800 3 journal journal_replay:
> applying op seq 4423
> -40> 2015-09-07 15:13:02.530088 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4423
> -39> 2015-09-07 15:13:02.530097 7f74cea0d800 2 -53> 2015-09-07
> 15:13:02.529857 7f74cea0d800 2 journal read_entry 85843968 : seq 4420 259
> bytes
> -52> 2015-09-07 15:13:02.529860 7f74cea0d800 3 journal journal_replay:
> applying op seq 4420
> -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2)
> No such file or directory
> -50> 2015-09-07 15:13:02.529946 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4420
> -49> 2015-09-07 15:13:02.529958 7f74cea0d800 2 journal read_entry
> 85848064 : seq 4421 46 bytes
> -48> 2015-09-07 15:13:02.529962 7f74cea0d800 3 journal journal_replay:
> applying op seq 4421
> -47> 2015-09-07 15:13:02.529967 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4421
> -46> 2015-09-07 15:13:02.529976 7f74cea0d800 2 journal read_entry
> 85852160 : seq 4422 259 bytes
> -45> 2015-09-07 15:13:02.529979 7f74cea0d800 3 journal journal_replay:
> applying op seq 4422
> -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2)
> No such file or directory
> -43> 2015-09-07 15:13:02.530070 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4422
> -42> 2015-09-07 15:13:02.530082 7f74cea0d800 2 journal read_entry
> 85856256 : seq 4423 46 bytes
> -41> 2015-09-07 15:13:02.530085 7f74cea0d800 3 journal journal_replay:
> applying op seq 4423
> -40> 2015-09-07 15:13:02.530088 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4423
> -39> 2015-09-07 15:13:02.530097 7f74cea0d800 2 journal read_entry
> 85860352 : seq 4424 959 bytes
> -38> 2015-09-07 15:13:02.530100 7f74cea0d800 3 journal journal_replay:
> applying op seq 4424
> -37> 2015-09-07 15:13:02.530135 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4424
> -38> 2015-09-07 15:13:02.530100 7f74cea0d800 3 journal journal_replay:
> applying op seq 4424
> -37> 2015-09-07 15:13:02.530135 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4424
> -36> 2015-09-07 15:13:02.530152 7f74cea0d800 2 journal read_entry
> 85864448 : seq 4425 959 bytes
> -36> 2015-09-07 15:13:02.530152 7f74cea0d800 2 journal read_entry
> 85864448 : seq 4425 959 bytes
> -35> 2015-09-07 15:13:02.530155 7f74cea0d800 3 -35> 2015-09-07
> 15:13:02.530155 7f74cea0d800 3 journal journal_replay: applying op seq 4425
> -34> 2015-09-07 15:13:02.530232 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4425journal journal_replay: r = 0, op_seq now 4425
> -33> 2015-09-07 15:13:02.530245 7f74cea0d800 2 journal read_entry
> 85868544 : seq 4426 259 bytes
> journal read_entry 85868544 : seq 4426 259 bytes
> -32> 2015-09-07 15:13:02.530251 7f74cea0d800 3 -32> 2015-09-07
> 15:13:02.530251 7f74cea0d800 3 journal journal_replay: applying op seq 4426
>
> -31> 2015-09-07 15:13:02.530273 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2)
> No such file or directoryfilestore(/var/lib/ceph/osd/ceph-0) could not find
> 1//head//3 in index: (2) No such file or directory
> -30> 2015-09-07 15:13:02.530299 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4426
> journal journal_replay: r = 0, op_seq now 4426
> -29> 2015-09-07 15:13:02.530310 7f74cea0d800 2 -29> 2015-09-07
> 15:13:02.530310 7f74cea0d800 2 journal read_entry 85872640 : seq 4427 46
> bytes
> -28> 2015-09-07 15:13:02.530312 7f74cea0d800 3 journal journal_replay:
> applying op seq 4427
> -27> 2015-09-07 15:13:02.530316 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4427
> -27> 2015-09-07 15:13:02.530316 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4427
> -26> 2015-09-07 15:13:02.530325 7f74cea0d800 2 journal read_entry
> 85876736 : seq 4428 259 bytes
> -26> 2015-09-07 15:13:02.530325 7f74cea0d800 2 journal read_entry
> 85876736 : seq 4428 259 bytes
> -25> 2015-09-07 15:13:02.530328 7f74cea0d800 3 journal journal_replay:
> applying op seq 4428
> -25> 2015-09-07 15:13:02.530328 7f74cea0d800 3 journal journal_replay:
> applying op seq 4428
> -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1 -24> 2015-09-07
> 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could
> not find 0//head//3 in index: (2) No such file or directory
> 0//head//3 in index: (2) No such file or directory
> -23> 2015-09-07 15:13:02.530389 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4428
> -22> 2015-09-07 15:13:02.530401 7f74cea0d800 2 journal read_entry
> 85880832 : seq 4429 46 bytes
>
> -21> 2015-09-07 15:13:02.530404 7f74cea0d800 3 -21> 2015-09-07
> 15:13:02.530404 7f74cea0d800 3 journal journal_replay: applying op seq 4429
>
> -20> 2015-09-07 15:13:02.530448 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4429journal journal_replay: r = 0, op_seq now 4429
> -19> 2015-09-07 15:13:02.530458 7f74cea0d800 2 journal read_entry
> 85884928 : seq 4430 259 bytes
> journal read_entry 85884928 : seq 4430 259 bytes
> -18> 2015-09-07 15:13:02.530465 7f74cea0d800 3 journal journal_replay:
> applying op seq 4430 -18> 2015-09-07 15:13:02.530465 7f74cea0d800 3
> journal journal_replay: applying op seq 4430
> -17> 2015-09-07 15:13:02.530487 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2)
> No such file or directory
> filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2)
> No such file or directory
> -16> 2015-09-07 15:13:02.530514 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4430
> -15> 2015-09-07 15:13:02.530525 7f74cea0d800 2 journal read_entry
> 85889024 : seq 4431 46 bytes
> -14> 2015-09-07 15:13:02.530528 7f74cea0d800 3 journal journal_replay:
> applying op seq 4431
> -13> 2015-09-07 15:13:02.530532 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4431
> -12> 2015-09-07 15:13:02.530541 7f74cea0d800 2 journal read_entry
> 85893120 : seq 4432 259 bytes
> -11> 2015-09-07 15:13:02.530549 7f74cea0d800 3 journal journal_replay:
> applying op seq 4432
> -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2)
> No such file or directory
> -9> 2015-09-07 15:13:02.530593 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4432
> -8> 2015-09-07 15:13:02.530604 7f74cea0d800 2 journal read_entry
> 85897216 : seq 4433 46 bytes
> -7> 2015-09-07 15:13:02.530606 7f74cea0d800 3 journal journal_replay:
> applying op seq 4433
> -6> 2015-09-07 15:13:02.530610 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4433
> -5> 2015-09-07 15:13:02.530621 7f74cea0d800 2 journal read_entry
> 85901312 : seq 4434 259 bytes
> -4> 2015-09-07 15:13:02.530624 7f74cea0d800 3 journal journal_replay:
> applying op seq 4434
> -3> 2015-09-07 15:13:02.530770 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not
> handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0)
> -2> 2015-09-07 15:13:02.530798 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd
> data dir
> -1> 2015-09-07 15:13:02.530801 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
> {
> "ops": [
> {
> "op_num": 0,
> "op_name": "remove",
> "collection": "3.5_head",
> "oid": "5\/\/head\/\/3"
> },
> {
> "op_num": 1,
> "op_name": "rmcoll",
> "collection": "3.5_head"
> }
> ]
> }
>
> 0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In
> function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&,
> uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07
> 15:13:02.530848
> os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")
>
> ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
> 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
> int, ThreadPool::TPHandle*)+0xc16) [0x975a06]
> 2: (FileStore::_do_transactions(std::list std::allocator >&, unsigned long,
> ThreadPool::TPHandle*)+0x64) [0x97d794]
> 3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
> 4: (FileStore::mount()+0x2d0e) [0x963d5e]
> 5: (OSD::init()+0x273) [0x6891f3]
> 6: (main()+0x384f) [0x62e10f]
> 7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
> 8: /usr/bin/ceph-osd() [0x629819]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd -16> 2015-09-07 15:13:02.530514 7f74cea0d800 3 journal
> journal_replay: r = 0, op_seq now 4430
> -15> 2015-09-07 15:13:02.530525 7f74cea0d800 2 journal read_entry
> 85889024 : seq 4431 46 bytes
> -14> 2015-09-07 15:13:02.530528 7f74cea0d800 3 journal journal_replay:
> applying op seq 4431
> -13> 2015-09-07 15:13:02.530532 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4431
> -12> 2015-09-07 15:13:02.530541 7f74cea0d800 2 journal read_entry
> 85893120 : seq 4432 259 bytes
> -11> 2015-09-07 15:13:02.530549 7f74cea0d800 3 journal journal_replay:
> applying op seq 4432
> -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1
> filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2)
> No such file or directory
> -9> 2015-09-07 15:13:02.530593 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4432
> -8> 2015-09-07 15:13:02.530604 7f74cea0d800 2 journal read_entry
> 85897216 : seq 4433 46 bytes
> -7> 2015-09-07 15:13:02.530606 7f74cea0d800 3 journal journal_replay:
> applying op seq 4433
> -6> 2015-09-07 15:13:02.530610 7f74cea0d800 3 journal journal_replay: r
> = 0, op_seq now 4433
> -5> 2015-09-07 15:13:02.530621 7f74cea0d800 2 journal read_entry
> 85901312 : seq 4434 259 bytes
> -4> 2015-09-07 15:13:02.530624 7f74cea0d800 3 journal journal_replay:
> applying op seq 4434
> -3> 2015-09-07 15:13:02.530770 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) error (39) Directory not empty not
> handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0)
> -2> 2015-09-07 15:13:02.530798 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd
> data dir
> -1> 2015-09-07 15:13:02.530801 7f74cea0d800 0
> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
> {
> "ops": [
> {
> "op_num": 0,
> "op_name": "remove",
> "collection": "3.5_head",
> "oid": "5\/\/head\/\/3"
> },
> {
> "op_num": 1,
> "op_name": "rmcoll",
> "collection": "3.5_head"
> }
> ]
> }
>
> 0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In
> function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&,
> uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07
> 15:13:02.530848
> os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")
>
> ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
> 1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
> int, ThreadPool::TPHandle*)+0xc16) [0x975a06]
> 2: (FileStore::_do_transactions(std::list std::allocator >&, unsigned long,
> ThreadPool::TPHandle*)+0x64) [0x97d794]
> 3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
> 4: (FileStore::mount()+0x2d0e) [0x963d5e]
> 5: (OSD::init()+0x273) [0x6891f3]
> 6: (main()+0x384f) [0x62e10f]
> 7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
> 8: /usr/bin/ceph-osd() [0x629819]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.0.log
> --- end dump of recent events ---
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.0.log
> --- end dump of recent events ---
> 2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) **
> in thread 7f74cea0d800
>
> ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
> 1: /usr/bin/ceph-osd() [0xa476a5]
> 2: /lib64/libpthread.so.0() [0x3c7100f710]
> 3: (gsignal()+0x35) [0x3c70c32625]
> 4: (abort()+0x175) [0x3c70c33e05]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> --- begin dump of recent events ---
> 0> 2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal
> (Aborted) **
> in thread 7f74cea0d800
>
> ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
> 1: /usr/bin/ceph-osd() [0xa476a5]
> 2: /lib64/libpthread.so.0() [0x3c7100f710]
> 3: (gsignal()+0x35) [0x3c70c32625]
> 4: (abort()+0x175) [0x3c70c33e05]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 keyvaluestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.0.log
> --- end dump of recent events ---
>
>
> I hope you could help me. Any advice is appreciated.
>
> Thanks,
> Aaron
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.0.2
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJV7wX6CRDmVDuy+mK58QAAfLwQAMYzI9ITbfvNnjiNtkV/
CsVtZr+7Ajgf9vTX+JdUCyTUxicdREgM/IfWbgKXlZ4F6Y1MN1tm1aq10gdj
G7yqxVc7q5aixjEhu8wC/p0P6sZv6FPp8i3jnssvxAd8TJ4QqkT1yqPiJyOw
6lnXKTpnE5EqMw+h99/XiEm2b8KNNbbA7QF+li0Mn7mNeG9GJhDhDVJEL9sG
V0v4UoBei2EU+Bw2cusaZ7lPrd1/H0T54Esc3GIIl7zGfkH8E4A68dYYsdCf
7Hd6HktgSl0+eTBWAI7/r8W2mUCTk7bXmyuv/NUh9mXaDOG9PyjlTxKk5jf4
x9xecDq1riAejyaCVGP6EYdSKlDRy0u0n3AXpbRx0cjU7Gv0DXKvmiWwrVLh
rRrXjyPRECedPUJMauDyCr83W2xrgn/cIQQx8NSc//+vcGrp/qUcyDtTYtkH
bG+ySdLDjYm6kKwesGohaHsq3qaGwux0iQ8nwd4z2JfDDbggiEbjr6Boypbq
owruQuI8AmgSxm6i9VEfoLhPt03N2Dhsgg8FUvcJ0CQzOgRBzgMN+RRmO5sT
VoD+3F44BHx6m46ZsBIOhfv1jr6m2IJdiw/HoMiFKlW2Wol3DUleILJWFzkL
QoKkP0qOXU4kXO/qF8PUR0YZkAI+3Q9K3t/jGuWZHSzm1SUsV4AS4yk/nBER
EZDN
=L6jU
-----END PGP SIGNATURE-----
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
