Re: [Problem] I cannot start the OSD daemon

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 


Supplement[On OSD-0 Node]:

[root@node1 ceph]# ls /var/lib/ceph/osd/ceph-0/current/
0.0_head   0.18_head  0.20_head  0.29_head  0.31_head  0.3a_head  0.6_head  0.f_head  2.0_head  commit_op_seq
0.10_head  0.19_head  0.21_head  0.2a_head  0.32_head  0.3b_head  0.7_head  1.0_head  2.1_head  meta
0.11_head  0.1a_head  0.22_head  0.2b_head  0.33_head  0.3c_head  0.8_head  1.1_head  2.2_head  nosnap
0.12_head  0.1b_head  0.23_head  0.2c_head  0.34_head  0.3d_head  0.9_head  1.2_head  2.3_head  omap
0.13_head  0.1c_head  0.24_head  0.2d_head  0.35_head  0.3e_head  0.a_head  1.3_head  2.4_head
0.14_head  0.1d_head  0.25_head  0.2e_head  0.36_head  0.3f_head  0.b_head  1.4_head  2.5_head
0.15_head  0.1e_head  0.26_head  0.2f_head  0.37_head  0.3_head   0.c_head  1.5_head  2.6_head
0.16_head  0.1f_head  0.27_head  0.2_head   0.38_head  0.4_head   0.d_head  1.6_head  2.7_head
0.17_head  0.1_head   0.28_head  0.30_head  0.39_head  0.5_head   0.e_head  1.7_head  3.5_head
[root@node1 ceph]#
[root@node1 ceph]# ls /var/lib/ceph/osd/ceph-0/current/3.5_head/
__head_BD49D10D__3
[root@node1 ceph]#
[root@node1 ceph]# ceph osd stat
     osdmap e20: 2 osds: 0 up, 1 in
[root@node1 ceph]#
[root@node1 ceph]# ceph osd tree
ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.21997 root default                                    
-2 0.10999     host node1                                  
 0 0.10999         osd.0     down        0          1.00000
-3 0.10999     host node2                                  
 1 0.10999         osd.1     down  1.00000          1.00000
[root@node1 ceph]#
[root@node1 ceph]# ceph osd dump
epoch 20
fsid 86130593-7041-4066-aa9b-963f4c5a6832
created 2015-08-05 15:42:28.938948
modified 2015-09-07 10:48:45.182566
flags
pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0
max_osd 2
osd.0 down out weight 0 up_from 4 up_thru 14 down_at 17 last_clean_interval [0,0) 10.241.226.116:6800/1084 10.241.226.116:6801/1084 10.241.226.116:6802/1084 10.241.226.116:6803/1084 autoout,exists c5f917c4-fd48-4bbf-954a-ab1ee1b524ad
osd.1 down in  weight 1 up_from 8 up_thru 17 down_at 20 last_clean_interval [0,0) 10.241.226.117:6800/1949 10.241.226.117:6801/1949 10.241.226.117:6802/1949 10.241.226.117:6803/1949 exists e1aaeab0-627c-4f22-b506-4856f2c8befb
[root@node1 ceph-0]# ls current/0.11_head/
__head_00000011__0
[root@node1 ceph]#


在 2015/9/7 16:16, Aaron 写道:
Hi All,

I cannot start the OSD daemon, and need your helps. Any advice is appreciated.

When I deployed the ceph cluster through ceph-deploy, it worked. But after some days, all the OSD daemons were down, and I could not start them. Then I redeployed it some times, and it was still this case.

The environment:
OS: CentOS 6.6
Ceph: hammer(0.92.2)

When the OSD daemon was down, I started the OSD daemon on the OSD node, and the console outputted:
[root@node1 ~]# service ceph start
=== osd.0 ===
libust[2285/2285]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
create-or-move updated item name 'osd.0' weight 0.11 at location {host=node1,root=default} to crush map
Starting Ceph osd.0 on node1...
libust[2329/2329]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
[root@node1 ~]#

And the log file of ceph, /var/log/ceph/ceph-osd.0.log, recorded:
2015-09-07 15:13:02.433546 7f74cea0d800  0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045
2015-09-07 15:13:02.508300 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53)
2015-09-07 15:13:02.511270 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2015-09-07 15:13:02.511293 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option

2015-09-07 15:13:02.520826 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported
2015-09-07 15:13:02.523890 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs
2015-09-07 15:13:02.527387 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2015-09-07 15:13:02.528890 7f74cea0d800  1 2015-09-07 15:13:02.528890 7f74cea0d800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0

2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory
2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory
2015-09-07 15:13:02.530273 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directory
2015-09-07 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2) No such file or directory

2015-09-07 15:13:02.530487 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory
2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory
2015-09-07 15:13:02.530770 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0)  error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0)
2015-09-07 15:13:02.530798 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir

2015-09-07 15:13:02.530801 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}

          "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}

2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06]
2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794]
3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
4: (FileStore::mount()+0x2d0e) [0x963d5e]
5: (OSD::init()+0x273) [0x6891f3]
6: (main()+0x384f) [0x62e10f]
7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
8: /usr/bin/ceph-osd() [0x629819]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -126> 2015-09-07 15:13:02.431001 7f74cea0d800  5 asok(0x5630000) register_command perfcounters_dump hook 0x55a8050
  -125> 2015-09-07 15:13:02.431054 7f74cea0d800  5 asok(0x5630000) register_command 1 hook 0x55a8050
  -126> 2015-09-07 15:13:02.431001 7f74cea0d800  5 asok(0x5630000) register_command perfcounters_dump hook 0x55a8050
  -125> 2015-09-07 15:13:02.431054 7f74cea0d800  5 asok(0x5630000) register_command 1 hook 0x55a8050
  -124> 2015-09-07 15:13:02.431059 7f74cea0d800  5 asok(0x5630000) register_command perf dump hook 0x55a8050  -124> 2015-09-07 15:13:02.431059 7f74cea0d800  5 asok(0x5630000) register_command perf dump hook 0x55a8050
  -123> 2015-09-07 15:13:02.431070 7f74cea0d800  5 asok(0x5630000) register_command perfcounters_schema hook 0x55a8050

  -122> 2015-09-07 15:13:02.431075 7f74cea0d800  5   -122> 2015-09-07 15:13:02.431075 7f74cea0d800  5 asok(0x5630000) register_command 2 hook 0x55a8050

  -121> 2015-09-07 15:13:02.431085 7f74cea0d800  5 asok(0x5630000) register_command perf schema hook 0x55a8050  -121> 2015-09-07 15:13:02.431085 7f74cea0d800  5 asok(0x5630000) register_command perf schema hook 0x55a8050
  -120> 2015-09-07 15:13:02.431089 7f74cea0d800  5 asok(0x5630000) register_command perf reset hook 0x55a8050
  -119> 2015-09-07 15:13:02.431097 7f74cea0d800  5 asok(0x5630000) register_command config show hook 0x55a8050
  -119> 2015-09-07 15:13:02.431097 7f74cea0d800  5 asok(0x5630000) register_command config show hook 0x55a8050
  -118> 2015-09-07 15:13:02.431101 7f74cea0d800  5 asok(0x5630000) register_command config set hook 0x55a8050
  -118> 2015-09-07 15:13:02.431101 7f74cea0d800  5 asok(0x5630000) register_command config set hook 0x55a8050
  -117> 2015-09-07 15:13:02.431109 7f74cea0d800  5 asok(0x5630000) register_command config get hook 0x55a8050
  -116> 2015-09-07 15:13:02.431112 7f74cea0d800  5 asok(0x5630000) register_command config diff hook 0x55a8050
  -115> 2015-09-07 15:13:02.431121 7f74cea0d800  5 asok(0x5630000) register_command log flush hook 0x55a8050
  -117> 2015-09-07 15:13:02.431109 7f74cea0d800  5 asok(0x5630000) register_command config get hook 0x55a8050
  -116> 2015-09-07 15:13:02.431112 7f74cea0d800  5 asok(0x5630000) register_command config diff hook 0x55a8050
  -115> 2015-09-07 15:13:02.431121 7f74cea0d800  5 asok(0x5630000) register_command log flush hook 0x55a8050
  -114> 2015-09-07 15:13:02.431124 7f74cea0d800  5 asok(0x5630000) register_command log dump hook 0x55a8050
asok(0x5630000) register_command log dump hook 0x55a8050
  -113> 2015-09-07 15:13:02.431133 7f74cea0d800  5 asok(0x5630000) register_command log reopen hook 0x55a8050
asok(0x5630000) register_command log reopen hook 0x55a8050
  -112> 2015-09-07 15:13:02.433546 7f74cea0d800  0   -112> 2015-09-07 15:13:02.433546 7f74cea0d800  0 ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3), process ceph-osd, pid 1045
  -111> 2015-09-07 15:13:02.434302 7f74cea0d800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/1045 need_addr=1accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/1045 need_addr=1
  -110> 2015-09-07 15:13:02.434355 7f74cea0d800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/1045 need_addr=1
  -109> 2015-09-07 15:13:02.434380 7f74cea0d800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/1045 need_addr=1accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/1045 need_addr=1
  -108> 2015-09-07 15:13:02.434403 7f74cea0d800  1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1
accepter.accepter.bind my_inst.addr is 0.0.0.0:6803/1045 need_addr=1
  -107> 2015-09-07 15:13:02.435349 7f74cea0d800  1 finished global_init_daemonize
  -107> 2015-09-07 15:13:02.435349 7f74cea0d800  1 finished global_init_daemonize
  -106> 2015-09-07 15:13:02.491120 7f74cea0d800  5 asok(0x5630000) init /var/run/ceph/ceph-osd.0.asok
  -106> 2015-09-07 15:13:02.491120 7f74cea0d800  5 asok(0x5630000) init /var/run/ceph/ceph-osd.0.asok
  -105> 2015-09-07 15:13:02.491164 7f74cea0d800  5 asok(0x5630000) bind_and_listen /var/run/ceph/ceph-osd.0.asok
  -105> 2015-09-07 15:13:02.491164 7f74cea0d800  5 asok(0x5630000) bind_and_listen /var/run/ceph/ceph-osd.0.asok
  -104> 2015-09-07 15:13:02.491404 7f74cea0d800  5 asok(0x5630000) register_command 0 hook 0x55a00b0
  -104> 2015-09-07 15:13:02.491404 7f74cea0d800  5 asok(0x5630000) register_command 0 hook 0x55a00b0
  -103> 2015-09-07 15:13:02.491429 7f74cea0d800  5 asok(0x5630000) register_command version hook 0x55a00b0
  -103> 2015-09-07 15:13:02.491429 7f74cea0d800  5 asok(0x5630000) register_command version hook 0x55a00b0
  -102> 2015-09-07 15:13:02.491435 7f74cea0d800  5 asok(0x5630000) register_command git_version hook 0x55a00b0  -102> 2015-09-07 15:13:02.491435 7f74cea0d800  5 asok(0x5630000) register_command git_version hook 0x55a00b0
  -101> 2015-09-07 15:13:02.491441 7f74cea0d800  5 f74cea0d800  5 asok(0x5630000) register_command help hook 0x55a8150
  -100> 2015-09-07 15:13:02.491445 7f74cea0d800  5 asok(0x5630000) register_command get_command_descriptions hook 0x55a8140
   -99> 2015-09-07 15:13:02.491545 7f74cea0d800 10 monclient(hunting): build_initial_monmap
   -98> 2015-09-07 15:13:02.491563 7f74cd141700  5 asok(0x5630000) entry start
   -97> 2015-09-07 15:13:02.506754 7f74cea0d800  5 adding auth protocol: cephx
   -96> 2015-09-07 15:13:02.506788 7f74cea0d800  5 adding auth protocol: cephx
   -95> 2015-09-07 15:13:02.507056 7f74cea0d800  5 asok(0x5630000) register_command objecter_requests hook 0x55a8180
   -94> 2015-09-07 15:13:02.507286 7f74cea0d800  1 -- 0.0.0.0:6800/1045 messenger.start
   -93> 2015-09-07 15:13:02.507367 7f74cea0d800  1 -- :/0 messenger.start
   -92> 2015-09-07 15:13:02.507504 7f74cea0d800  1 -- 0.0.0.0:6803/1045 messenger.start
   -91> 2015-09-07 15:13:02.507637 7f74cea0d800  1 -- 0.0.0.0:6802/1045 messenger.start
   -90> 2015-09-07 15:13:02.507744 7f74cea0d800  1 -- 0.0.0.0:6801/1045 messenger.start
   -89> 2015-09-07 15:13:02.507889 7f74cea0d800  1 -- :/0 messenger.start
   -88> 2015-09-07 15:13:02.508178 7f74cea0d800  2 osd.0 0 mounting /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
   -87> 2015-09-07 15:13:02.508300 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0xef53)
   -86> 2015-09-07 15:13:02.511270 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
   -85> 2015-09-07 15:13:02.511293 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
   -85> 2015-09-07 15:13:02.511293 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
   -84> 2015-09-07 15:13:02.520826 7f74cea0d800  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syscall(SYS_syncfs, fd) fully supported

   -83> 2015-09-07 15:13:02.523890 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrsfilestore(/var/lib/ceph/osd/ceph-0) limited size xattrs
   -82> 2015-09-07 15:13:02.527387 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
   -81> 2015-09-07 15:13:02.528819 7f74cea0d800  2    -81> 2015-09-07 15:13:02.528819 7f74cea0d800  2 journal open /var/lib/ceph/osd/ceph-0/journal fsid c5f917c4-fd48-4bbf-954a-ab1ee1b524ad fs_op_seq 4411
   -80> 2015-09-07 15:13:02.528870 7f74cea0d800 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway

   -79> 2015-09-07 15:13:02.528890 7f74cea0d800  1    -79> 2015-09-07 15:13:02.528890 7f74cea0d800  1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 19: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
   -78> 2015-09-07 15:13:02.528954 7f74cea0d800  2 journal read_entry 85811200 : seq 4412 3703 bytes
journal read_entry 85811200 : seq 4412 3703 bytes
   -77> 2015-09-07 15:13:02.528966 7f74cea0d800  2 journal read_entry 85811200 : seq 4412 3703 bytes
   -77> 2015-09-07 15:13:02.528966 7f74cea0d800  2 journal read_entry 85811200 : seq 4412 3703 bytes
   -76> 2015-09-07 15:13:02.528971 7f74cea0d800  3    -76> 2015-09-07 15:13:02.528971 7f74cea0d800  3 journal journal_replay: applying op seq 4412
   -75> 2015-09-07 15:13:02.529399 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4412journal journal_replay: r = 0, op_seq now 4412
   -74> 2015-09-07 15:13:02.529432 7f74cea0d800  2 journal read_entry 85815296 : seq 4413 959 bytes
journal read_entry 85815296 : seq 4413 959 bytes
   -73> 2015-09-07 15:13:02.529436 7f74cea0d800  3 journal journal_replay: applying op seq 4413
   -73> 2015-09-07 15:13:02.529436 7f74cea0d800  3 journal journal_replay: applying op seq 4413
   -72> 2015-09-07 15:13:02.529476 7f74cea0d800  3    -72> 2015-09-07 15:13:02.529476 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4413
   -71> 2015-09-07 15:13:02.529501 7f74cea0d800  2 journal read_entry 85819392 : seq 4414 46 bytes
   -70> 2015-09-07 15:13:02.529573 7f74cea0d800  3    -70> 2015-09-07 15:13:02.529573 7f74cea0d800  3 journal journal_replay: applying op seq 4414

   -69> 2015-09-07 15:13:02.529579 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4414   -69> 2015-09-07 15:13:02.529579 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4414
   -68> 2015-09-07 15:13:02.529590 7f74cea0d800  2 journal read_entry 85823488 : seq 4415 959 bytes
journal read_entry 85823488 : seq 4415 959 bytes
   -67> 2015-09-07 15:13:02.529592 7f74cea0d800  3    -67> 2015-09-07 15:13:02.529592 7f74cea0d800  3 journal journal_replay: applying op seq 4415
   -66> 2015-09-07 15:13:02.529620 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4415
   -65> 2015-09-07 15:13:02.529645 7f74cea0d800  2    -65> 2015-09-07 15:13:02.529645 7f74cea0d800  2 journal read_entry 85827584 : seq 4416 959 bytes

   -64> 2015-09-07 15:13:02.529648 7f74cea0d800  3 journal journal_replay: applying op   -64> 2015-09-07 15:13:02.529648 7f74cea0d800  3 journal journal_replay: applying op seq 4416 seq 4416
   -63> 2015-09-07 15:13:02.529672 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4416
   -62> 2015-09-07 15:13:02.529687 7f74cea0d800  2 journal read_entry 85831680 : seq 4417 959 bytes
   -61> 2015-09-07 15:13:02.529690 7f74cea0d800  3 journal journal_replay: applying op seq 4417   -61> 2015-09-07 15:13:02.529690 7f74cea0d800  3 journal journal_replay: applying op seq 4417
   -60> 2015-09-07 15:13:02.529711 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4417
journal journal_replay: r = 0, op_seq now 4417
   -59> 2015-09-07 15:13:02.529776 7f74cea0d800  2    -59> 2015-09-07 15:13:02.529776 7f74cea0d800  2 journal read_entry 85835776 : seq 4418 959 bytes
   -58> 2015-09-07 15:13:02.529779 7f74cea0d800  3 journal journal_replay: applying op seq 4418
journal journal_replay: applying op seq 4418
   -57> 2015-09-07 15:13:02.529803 7f74cea0d800  3    -57> 2015-09-07 15:13:02.529803 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4418
   -56> 2015-09-07 15:13:02.529819 7f74cea0d800  2 journal read_entry 85839872 : seq 4419 959 bytesjournal read_entry 85839872 : seq 4419 959 bytes
   -55> 2015-09-07 15:13:02.529822 7f74cea0d800  3 journal journal_replay: applying op seq 4419
journal journal_replay: applying op seq 4419
   -54> 2015-09-07 15:13:02.529842 7f74cea0d800  3    -54> 2015-09-07 15:13:02.529842 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4419
   -53> 2015-09-07 15:13:02.529857 7f74cea0d800  2 journal read_entry 85843968 : seq 4420 259 bytes
   -52> 2015-09-07 15:13:02.529860 7f74cea0d800  3 journal journal_replay: applying op seq 4420
   -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory
   -50> 2015-09-07 15:13:02.529946 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4420
   -49> 2015-09-07 15:13:02.529958 7f74cea0d800  2 journal read_entry 85848064 : seq 4421 46 bytes
   -48> 2015-09-07 15:13:02.529962 7f74cea0d800  3 journal journal_replay: applying op seq 4421
   -47> 2015-09-07 15:13:02.529967 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4421
   -46> 2015-09-07 15:13:02.529976 7f74cea0d800  2 journal read_entry 85852160 : seq 4422 259 bytes
   -45> 2015-09-07 15:13:02.529979 7f74cea0d800  3 journal journal_replay: applying op seq 4422
   -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory
   -43> 2015-09-07 15:13:02.530070 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4422
   -42> 2015-09-07 15:13:02.530082 7f74cea0d800  2 journal read_entry 85856256 : seq 4423 46 bytes
   -41> 2015-09-07 15:13:02.530085 7f74cea0d800  3 journal journal_replay: applying op seq 4423
   -40> 2015-09-07 15:13:02.530088 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4423
   -39> 2015-09-07 15:13:02.530097 7f74cea0d800  2    -53> 2015-09-07 15:13:02.529857 7f74cea0d800  2 journal read_entry 85843968 : seq 4420 259 bytes
   -52> 2015-09-07 15:13:02.529860 7f74cea0d800  3 journal journal_replay: applying op seq 4420
   -51> 2015-09-07 15:13:02.529881 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 3//head//3 in index: (2) No such file or directory
   -50> 2015-09-07 15:13:02.529946 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4420
   -49> 2015-09-07 15:13:02.529958 7f74cea0d800  2 journal read_entry 85848064 : seq 4421 46 bytes
   -48> 2015-09-07 15:13:02.529962 7f74cea0d800  3 journal journal_replay: applying op seq 4421
   -47> 2015-09-07 15:13:02.529967 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4421
   -46> 2015-09-07 15:13:02.529976 7f74cea0d800  2 journal read_entry 85852160 : seq 4422 259 bytes
   -45> 2015-09-07 15:13:02.529979 7f74cea0d800  3 journal journal_replay: applying op seq 4422
   -44> 2015-09-07 15:13:02.530039 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 2//head//3 in index: (2) No such file or directory
   -43> 2015-09-07 15:13:02.530070 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4422
   -42> 2015-09-07 15:13:02.530082 7f74cea0d800  2 journal read_entry 85856256 : seq 4423 46 bytes
   -41> 2015-09-07 15:13:02.530085 7f74cea0d800  3 journal journal_replay: applying op seq 4423
   -40> 2015-09-07 15:13:02.530088 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4423
   -39> 2015-09-07 15:13:02.530097 7f74cea0d800  2 journal read_entry 85860352 : seq 4424 959 bytes
   -38> 2015-09-07 15:13:02.530100 7f74cea0d800  3 journal journal_replay: applying op seq 4424
   -37> 2015-09-07 15:13:02.530135 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4424
   -38> 2015-09-07 15:13:02.530100 7f74cea0d800  3 journal journal_replay: applying op seq 4424
   -37> 2015-09-07 15:13:02.530135 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4424
   -36> 2015-09-07 15:13:02.530152 7f74cea0d800  2 journal read_entry 85864448 : seq 4425 959 bytes
   -36> 2015-09-07 15:13:02.530152 7f74cea0d800  2 journal read_entry 85864448 : seq 4425 959 bytes
   -35> 2015-09-07 15:13:02.530155 7f74cea0d800  3    -35> 2015-09-07 15:13:02.530155 7f74cea0d800  3 journal journal_replay: applying op seq 4425
   -34> 2015-09-07 15:13:02.530232 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4425journal journal_replay: r = 0, op_seq now 4425
   -33> 2015-09-07 15:13:02.530245 7f74cea0d800  2 journal read_entry 85868544 : seq 4426 259 bytes
journal read_entry 85868544 : seq 4426 259 bytes
   -32> 2015-09-07 15:13:02.530251 7f74cea0d800  3    -32> 2015-09-07 15:13:02.530251 7f74cea0d800  3 journal journal_replay: applying op seq 4426

   -31> 2015-09-07 15:13:02.530273 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directoryfilestore(/var/lib/ceph/osd/ceph-0) could not find 1//head//3 in index: (2) No such file or directory
   -30> 2015-09-07 15:13:02.530299 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4426
journal journal_replay: r = 0, op_seq now 4426
   -29> 2015-09-07 15:13:02.530310 7f74cea0d800  2    -29> 2015-09-07 15:13:02.530310 7f74cea0d800  2 journal read_entry 85872640 : seq 4427 46 bytes
   -28> 2015-09-07 15:13:02.530312 7f74cea0d800  3 journal journal_replay: applying op seq 4427
   -27> 2015-09-07 15:13:02.530316 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4427
   -27> 2015-09-07 15:13:02.530316 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4427
   -26> 2015-09-07 15:13:02.530325 7f74cea0d800  2 journal read_entry 85876736 : seq 4428 259 bytes
   -26> 2015-09-07 15:13:02.530325 7f74cea0d800  2 journal read_entry 85876736 : seq 4428 259 bytes
   -25> 2015-09-07 15:13:02.530328 7f74cea0d800  3 journal journal_replay: applying op seq 4428
   -25> 2015-09-07 15:13:02.530328 7f74cea0d800  3 journal journal_replay: applying op seq 4428
   -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1    -24> 2015-09-07 15:13:02.530349 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 0//head//3 in index: (2) No such file or directory
0//head//3 in index: (2) No such file or directory
   -23> 2015-09-07 15:13:02.530389 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4428
   -22> 2015-09-07 15:13:02.530401 7f74cea0d800  2 journal read_entry 85880832 : seq 4429 46 bytes

   -21> 2015-09-07 15:13:02.530404 7f74cea0d800  3    -21> 2015-09-07 15:13:02.530404 7f74cea0d800  3 journal journal_replay: applying op seq 4429

   -20> 2015-09-07 15:13:02.530448 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4429journal journal_replay: r = 0, op_seq now 4429
   -19> 2015-09-07 15:13:02.530458 7f74cea0d800  2 journal read_entry 85884928 : seq 4430 259 bytes
journal read_entry 85884928 : seq 4430 259 bytes
   -18> 2015-09-07 15:13:02.530465 7f74cea0d800  3 journal journal_replay: applying op seq 4430   -18> 2015-09-07 15:13:02.530465 7f74cea0d800  3 journal journal_replay: applying op seq 4430
   -17> 2015-09-07 15:13:02.530487 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory
filestore(/var/lib/ceph/osd/ceph-0) could not find 7//head//3 in index: (2) No such file or directory
   -16> 2015-09-07 15:13:02.530514 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4430
   -15> 2015-09-07 15:13:02.530525 7f74cea0d800  2 journal read_entry 85889024 : seq 4431 46 bytes
   -14> 2015-09-07 15:13:02.530528 7f74cea0d800  3 journal journal_replay: applying op seq 4431
   -13> 2015-09-07 15:13:02.530532 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4431
   -12> 2015-09-07 15:13:02.530541 7f74cea0d800  2 journal read_entry 85893120 : seq 4432 259 bytes
   -11> 2015-09-07 15:13:02.530549 7f74cea0d800  3 journal journal_replay: applying op seq 4432
   -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory
    -9> 2015-09-07 15:13:02.530593 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4432
    -8> 2015-09-07 15:13:02.530604 7f74cea0d800  2 journal read_entry 85897216 : seq 4433 46 bytes
    -7> 2015-09-07 15:13:02.530606 7f74cea0d800  3 journal journal_replay: applying op seq 4433
    -6> 2015-09-07 15:13:02.530610 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4433
    -5> 2015-09-07 15:13:02.530621 7f74cea0d800  2 journal read_entry 85901312 : seq 4434 259 bytes
    -4> 2015-09-07 15:13:02.530624 7f74cea0d800  3 journal journal_replay: applying op seq 4434
    -3> 2015-09-07 15:13:02.530770 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0)  error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0)
    -2> 2015-09-07 15:13:02.530798 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir
    -1> 2015-09-07 15:13:02.530801 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}

     0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06]
2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794]
3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
4: (FileStore::mount()+0x2d0e) [0x963d5e]
5: (OSD::init()+0x273) [0x6891f3]
6: (main()+0x384f) [0x62e10f]
7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
8: /usr/bin/ceph-osd() [0x629819]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd   -16> 2015-09-07 15:13:02.530514 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4430
   -15> 2015-09-07 15:13:02.530525 7f74cea0d800  2 journal read_entry 85889024 : seq 4431 46 bytes
   -14> 2015-09-07 15:13:02.530528 7f74cea0d800  3 journal journal_replay: applying op seq 4431
   -13> 2015-09-07 15:13:02.530532 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4431
   -12> 2015-09-07 15:13:02.530541 7f74cea0d800  2 journal read_entry 85893120 : seq 4432 259 bytes
   -11> 2015-09-07 15:13:02.530549 7f74cea0d800  3 journal journal_replay: applying op seq 4432
   -10> 2015-09-07 15:13:02.530569 7f74cea0d800 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 6//head//3 in index: (2) No such file or directory
    -9> 2015-09-07 15:13:02.530593 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4432
    -8> 2015-09-07 15:13:02.530604 7f74cea0d800  2 journal read_entry 85897216 : seq 4433 46 bytes
    -7> 2015-09-07 15:13:02.530606 7f74cea0d800  3 journal journal_replay: applying op seq 4433
    -6> 2015-09-07 15:13:02.530610 7f74cea0d800  3 journal journal_replay: r = 0, op_seq now 4433
    -5> 2015-09-07 15:13:02.530621 7f74cea0d800  2 journal read_entry 85901312 : seq 4434 259 bytes
    -4> 2015-09-07 15:13:02.530624 7f74cea0d800  3 journal journal_replay: applying op seq 4434
    -3> 2015-09-07 15:13:02.530770 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0)  error (39) Directory not empty not handled on operation 0x55d8ef6 (4434.0.1, or op 1, counting from 0)
    -2> 2015-09-07 15:13:02.530798 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0) ENOTEMPTY suggests garbage data in osd data dir
    -1> 2015-09-07 15:13:02.530801 7f74cea0d800  0 filestore(/var/lib/ceph/osd/ceph-0)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "3.5_head",
            "oid": "5\/\/head\/\/3"
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "3.5_head"
        }
    ]
}

     0> 2015-09-07 15:13:02.533771 7f74cea0d800 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f74cea0d800 time 2015-09-07 15:13:02.530848
os/FileStore.cc: 2757: FAILED assert(0 == "unexpected error")

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xc16) [0x975a06]
2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x97d794]
3: (JournalingObjectStore::journal_replay(unsigned long)+0x61e) [0xa2927e]
4: (FileStore::mount()+0x2d0e) [0x963d5e]
5: (OSD::init()+0x273) [0x6891f3]
6: (main()+0x384f) [0x62e10f]
7: (__libc_start_main()+0xfd) [0x3c70c1ed5d]
8: /usr/bin/ceph-osd() [0x629819]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---
2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) **
in thread 7f74cea0d800

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: /usr/bin/ceph-osd() [0xa476a5]
2: /lib64/libpthread.so.0() [0x3c7100f710]
3: (gsignal()+0x35) [0x3c70c32625]
4: (abort()+0x175) [0x3c70c33e05]
5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2015-09-07 15:13:02.540432 7f74cea0d800 -1 *** Caught signal (Aborted) **
in thread 7f74cea0d800

ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: /usr/bin/ceph-osd() [0xa476a5]
2: /lib64/libpthread.so.0() [0x3c7100f710]
3: (gsignal()+0x35) [0x3c70c32625]
4: (abort()+0x175) [0x3c70c33e05]
5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x39930bea7d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.0.log
--- end dump of recent events ---


I hope you could help me. Any advice is appreciated.

Thanks,
Aaron





_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux