mds laggy or crashed

[Gagandeep Arora <aroragagan24@xxxxxxxxx>]

Hello, 

We are running ceph-0.67.4 with two mds and both of the mds daemons are crashing see the logs below:

[root@ceph1 ~]# ceph health detail

HEALTH_ERR mds rank 1 has failed; mds cluster is degraded; mds a is laggy

mds.1 has failed

mds cluster is degraded

mds.a at 192.168.6.101:6808/14609 rank 0 is replaying journal

mds.a at 192.168.6.101:6808/14609 is laggy/unresponsive

[root@ceph1 ~]# ceph mds dump

dumped mdsmap epoch 19386

epoch 19386

flags 0

created 2013-03-20 08:56:13.873024

modified 2013-10-22 11:58:31.374700

tableserver 0

root 0

session_timeout 60

session_autoclose 300

last_failure 19253

last_failure_osd_epoch 6648

compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding}

max_mds 2

in 0,1

up {0=222230}

failed 1

stopped

data_pools 0,13,14

metadata_pool 1

222230: 192.168.6.101:6808/14609 'a' mds.0.19 up:replay seq 1 laggy since 2013-10-22 11:55:50.972032

[root@ceph1 ~]# ceph-mds -i a -d

2013-10-22 11:55:28.093342 7f343195f7c0  0 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-mds, pid 14609

starting mds.a at :/0

2013-10-22 11:55:31.550871 7f342c593700  1 mds.-1.0 handle_mds_map standby

2013-10-22 11:55:32.151652 7f342c593700  1 mds.0.19 handle_mds_map i am now mds.0.19

2013-10-22 11:55:32.151658 7f342c593700  1 mds.0.19 handle_mds_map state change up:standby --> up:replay

2013-10-22 11:55:32.151661 7f342c593700  1 mds.0.19 replay_start

2013-10-22 11:55:32.151673 7f342c593700  1 mds.0.19  recovery set is 1

2013-10-22 11:55:32.151675 7f342c593700  1 mds.0.19  need osdmap epoch 6648, have 6647

2013-10-22 11:55:32.151677 7f342c593700  1 mds.0.19  waiting for osdmap 6648 (which blacklists prior instance)

2013-10-22 11:55:32.275413 7f342c593700  0 mds.0.cache creating system inode with ino:100

2013-10-22 11:55:32.275720 7f342c593700  0 mds.0.cache creating system inode with ino:1

mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f3428078700 time 2013-10-22 11:55:37.562600

mds/journal.cc: 1096: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x399d) [0x65b0ad]

 2: (EUpdate::replay(MDS*)+0x3a) [0x663c0a]

 3: (MDLog::_replay_thread()+0x5cf) [0x82e17f]

 4: (MDLog::ReplayThread::entry()+0xd) [0x6393ad]

 5: (()+0x7d15) [0x7f3430fc2d15]

 6: (clone()+0x6d) [0x7f342fa3948d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2013-10-22 11:55:37.563382 7f3428078700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f3428078700 time 2013-10-22 11:55:37.562600

mds/journal.cc: 1096: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x399d) [0x65b0ad]

 2: (EUpdate::replay(MDS*)+0x3a) [0x663c0a]

 3: (MDLog::_replay_thread()+0x5cf) [0x82e17f]

 4: (MDLog::ReplayThread::entry()+0xd) [0x6393ad]

 5: (()+0x7d15) [0x7f3430fc2d15]

 6: (clone()+0x6d) [0x7f342fa3948d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

  -386> 2013-10-22 11:55:28.090574 7f343195f7c0  5 asok(0x2780000) register_command perfcounters_dump hook 0x2748010

  -385> 2013-10-22 11:55:28.090635 7f343195f7c0  5 asok(0x2780000) register_command 1 hook 0x2748010

  -384> 2013-10-22 11:55:28.090640 7f343195f7c0  5 asok(0x2780000) register_command perf dump hook 0x2748010

  -383> 2013-10-22 11:55:28.090650 7f343195f7c0  5 asok(0x2780000) register_command perfcounters_schema hook 0x2748010

  -382> 2013-10-22 11:55:28.090658 7f343195f7c0  5 asok(0x2780000) register_command 2 hook 0x2748010

  -381> 2013-10-22 11:55:28.090665 7f343195f7c0  5 asok(0x2780000) register_command perf schema hook 0x2748010

  -380> 2013-10-22 11:55:28.090673 7f343195f7c0  5 asok(0x2780000) register_command config show hook 0x2748010

  -379> 2013-10-22 11:55:28.090698 7f343195f7c0  5 asok(0x2780000) register_command config set hook 0x2748010

  -378> 2013-10-22 11:55:28.090706 7f343195f7c0  5 asok(0x2780000) register_command config get hook 0x2748010

  -377> 2013-10-22 11:55:28.090712 7f343195f7c0  5 asok(0x2780000) register_command log flush hook 0x2748010

  -376> 2013-10-22 11:55:28.090718 7f343195f7c0  5 asok(0x2780000) register_command log dump hook 0x2748010

  -375> 2013-10-22 11:55:28.090726 7f343195f7c0  5 asok(0x2780000) register_command log reopen hook 0x2748010

  -374> 2013-10-22 11:55:28.093342 7f343195f7c0  0 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-mds, pid 14609

  -373> 2013-10-22 11:55:28.093910 7f343195f7c0  1 -- 192.168.6.101:0/0 learned my addr 192.168.6.101:0/0

  -372> 2013-10-22 11:55:28.093931 7f343195f7c0  1 accepter.accepter.bind my_inst.addr is 192.168.6.101:6808/14609 need_addr=0

  -371> 2013-10-22 11:55:28.096961 7f343195f7c0  5 asok(0x2780000) init /var/run/ceph/ceph-mds.a.asok

  -370> 2013-10-22 11:55:28.096984 7f343195f7c0  5 asok(0x2780000) bind_and_listen /var/run/ceph/ceph-mds.a.asok

  -369> 2013-10-22 11:55:28.097170 7f343195f7c0  5 asok(0x2780000) register_command 0 hook 0x27400b8

  -368> 2013-10-22 11:55:28.097184 7f343195f7c0  5 asok(0x2780000) register_command version hook 0x27400b8

  -367> 2013-10-22 11:55:28.097190 7f343195f7c0  5 asok(0x2780000) register_command git_version hook 0x27400b8

  -366> 2013-10-22 11:55:28.097194 7f343195f7c0  5 asok(0x2780000) register_command help hook 0x27480d0

  -365> 2013-10-22 11:55:28.097197 7f343195f7c0  5 asok(0x2780000) register_command get_command_descriptions hook 0x27480c0

  -364> 2013-10-22 11:55:28.097248 7f343195f7c0 10 monclient(hunting): build_initial_monmap

  -363> 2013-10-22 11:55:28.097252 7f342d595700  5 asok(0x2780000) entry start

  -362> 2013-10-22 11:55:28.097321 7f343195f7c0  1 -- 192.168.6.101:6808/14609 messenger.start

  -361> 2013-10-22 11:55:28.097381 7f343195f7c0  5 adding auth protocol: cephx

  -360> 2013-10-22 11:55:28.097391 7f343195f7c0  5 adding auth protocol: cephx

  -359> 2013-10-22 11:55:28.097511 7f343195f7c0  1 accepter.accepter.start

  -358> 2013-10-22 11:55:28.097538 7f343195f7c0 10 monclient(hunting): init

  -357> 2013-10-22 11:55:28.097546 7f343195f7c0  5 adding auth protocol: cephx

  -356> 2013-10-22 11:55:28.097549 7f343195f7c0 10 monclient(hunting): auth_supported 2 method cephx

  -355> 2013-10-22 11:55:28.097728 7f343195f7c0  2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-a/keyring

  -354> 2013-10-22 11:55:28.097802 7f343195f7c0 10 monclient(hunting): _reopen_session rank -1 name 

  -353> 2013-10-22 11:55:28.097879 7f343195f7c0 10 monclient(hunting): picked mon.b con 0x27b8f20 addr 192.168.6.102:6789/0

  -352> 2013-10-22 11:55:28.097895 7f343195f7c0 10 monclient(hunting): _send_mon_message to mon.b at 192.168.6.102:6789/0

  -351> 2013-10-22 11:55:28.097905 7f343195f7c0  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x27786c0 con 0x27b8f20

  -350> 2013-10-22 11:55:28.097922 7f343195f7c0 10 monclient(hunting): renew_subs

  -349> 2013-10-22 11:55:28.098682 7f342c593700  5 mds.-1.0 ms_handle_connect on 192.168.6.102:6789/0

  -348> 2013-10-22 11:55:31.097877 7f342b591700 10 monclient(hunting): tick

  -347> 2013-10-22 11:55:31.097904 7f342b591700  1 monclient(hunting): continuing hunt

  -346> 2013-10-22 11:55:31.097907 7f342b591700 10 monclient(hunting): _reopen_session rank -1 name 

  -345> 2013-10-22 11:55:31.097914 7f342b591700  1 -- 192.168.6.101:6808/14609 mark_down 0x27b8f20 -- 0x2850500

  -344> 2013-10-22 11:55:31.098005 7f342a58f700  2 -- 192.168.6.101:6808/14609 >> 192.168.6.102:6789/0 pipe(0x2850500 sd=7 :41444 s=4 pgs=11503 cs=1 l=1 c=0x27b8f20).reader couldn't read tag, Success

  -343> 2013-10-22 11:55:31.098060 7f342b591700 10 monclient(hunting): picked mon.c con 0x27b9600 addr 192.168.6.103:6789/0

  -342> 2013-10-22 11:55:31.098084 7f342b591700 10 monclient(hunting): _send_mon_message to mon.c at 192.168.6.103:6789/0

  -341> 2013-10-22 11:55:31.098072 7f342a58f700  2 -- 192.168.6.101:6808/14609 >> 192.168.6.102:6789/0 pipe(0x2850500 sd=7 :41444 s=4 pgs=11503 cs=1 l=1 c=0x27b8f20).fault 0: Success

  -340> 2013-10-22 11:55:31.098103 7f342b591700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x27786c0 con 0x27b9600

  -339> 2013-10-22 11:55:31.098216 7f342b591700 10 monclient(hunting): renew_subs

  -338> 2013-10-22 11:55:31.099151 7f342c593700  5 mds.-1.0 ms_handle_connect on 192.168.6.103:6789/0

  -337> 2013-10-22 11:55:31.099633 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 1 ==== mon_map v1 ==== 473+0+0 (1346944852 0 0) 0x28683c0 con 0x27b9600

  -336> 2013-10-22 11:55:31.099679 7f342c593700 10 monclient(hunting): handle_monmap mon_map v1

  -335> 2013-10-22 11:55:31.099700 7f342c593700 10 monclient(hunting):  got monmap 3, mon.c is now rank 2

  -334> 2013-10-22 11:55:31.099707 7f342c593700 10 monclient(hunting): dump:

epoch 3

fsid c0459c67-e2cd-45f7-b580-dec1afc9dea5

last_changed 2013-06-26 13:53:26.157985

created 2013-03-20 08:54:46.921197

0: 192.168.6.101:6789/0 mon.a

1: 192.168.6.102:6789/0 mon.b

2: 192.168.6.103:6789/0 mon.c

  -333> 2013-10-22 11:55:31.099754 7f342c593700  1 monclient(hunting): found mon.c

  -332> 2013-10-22 11:55:31.099766 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (1841418499 0 0) 0x27a1400 con 0x27b9600

  -331> 2013-10-22 11:55:31.099793 7f342c593700 10 monclient: my global_id is 222230

  -330> 2013-10-22 11:55:31.100009 7f342c593700 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -329> 2013-10-22 11:55:31.100023 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2778900 con 0x27b9600

  -328> 2013-10-22 11:55:31.100625 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (4272342048 0 0) 0x27a1800 con 0x27b9600

  -327> 2013-10-22 11:55:31.100728 7f342c593700 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -326> 2013-10-22 11:55:31.100739 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2778d80 con 0x27b9600

  -325> 2013-10-22 11:55:31.101547 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (220560408 0 0) 0x27a1600 con 0x27b9600

  -324> 2013-10-22 11:55:31.101659 7f342c593700 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -323> 2013-10-22 11:55:31.101670 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x28201c0 con 0x27b9600

  -322> 2013-10-22 11:55:31.101700 7f342c593700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-10-22 11:55:01.101700)

  -321> 2013-10-22 11:55:31.101717 7f342c593700 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -320> 2013-10-22 11:55:31.101725 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2778b40 con 0x27b9600

  -319> 2013-10-22 11:55:31.101750 7f343195f7c0  5 monclient: authenticate success, global_id 222230

  -318> 2013-10-22 11:55:31.101767 7f343195f7c0 10 monclient: wait_auth_rotating waiting (until 2013-10-22 11:56:01.101765)

  -317> 2013-10-22 11:55:31.102299 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 5 ==== mon_map v1 ==== 473+0+0 (1346944852 0 0) 0x28685a0 con 0x27b9600

  -316> 2013-10-22 11:55:31.102320 7f342c593700 10 monclient: handle_monmap mon_map v1

  -315> 2013-10-22 11:55:31.102337 7f342c593700 10 monclient:  got monmap 3, mon.c is now rank 2

  -314> 2013-10-22 11:55:31.102347 7f342c593700 10 monclient: dump:

epoch 3

fsid c0459c67-e2cd-45f7-b580-dec1afc9dea5

last_changed 2013-06-26 13:53:26.157985

created 2013-03-20 08:54:46.921197

0: 192.168.6.101:6789/0 mon.a

1: 192.168.6.102:6789/0 mon.b

2: 192.168.6.103:6789/0 mon.c

  -313> 2013-10-22 11:55:31.102391 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2221013279 0 0) 0x2820540 con 0x27b9600

  -312> 2013-10-22 11:55:31.102407 7f342c593700 10 monclient: handle_subscribe_ack sent 2013-10-22 11:55:28.097925 renew after 2013-10-22 11:57:58.097925

  -311> 2013-10-22 11:55:31.102430 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1277694908 0 0) 0x27a1a00 con 0x27b9600

  -310> 2013-10-22 11:55:31.102490 7f342c593700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-10-22 11:55:01.102489)

  -309> 2013-10-22 11:55:31.102510 7f343195f7c0 10 monclient: wait_auth_rotating done

  -308> 2013-10-22 11:55:31.102531 7f343195f7c0  5 asok(0x2780000) register_command objecter_requests hook 0x2748120

  -307> 2013-10-22 11:55:31.102601 7f343195f7c0 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -306> 2013-10-22 11:55:31.102612 7f343195f7c0  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- mdsbeacon(222230/a up:boot seq 1 v0) v2 -- ?+0 0x2838dc0 con 0x27b9600

  -305> 2013-10-22 11:55:31.102642 7f343195f7c0 10 monclient: renew_subs

  -304> 2013-10-22 11:55:31.102651 7f343195f7c0 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -303> 2013-10-22 11:55:31.102662 7f343195f7c0  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- mon_subscribe({monmap=4+,osdmap=0}) v2 -- ?+0 0x2820a80 con 0x27b9600

  -302> 2013-10-22 11:55:31.102685 7f343195f7c0 10 monclient: renew_subs

  -301> 2013-10-22 11:55:31.102692 7f343195f7c0 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -300> 2013-10-22 11:55:31.102702 7f343195f7c0  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- mon_subscribe({mdsmap=0+,monmap=4+,osdmap=0}) v2 -- ?+0 0x28208c0 con 0x27b9600

  -299> 2013-10-22 11:55:31.103849 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 8 ==== osd_map(6647..6647 src has 6100..6647) v3 ==== 26502+0+0 (496991870 0 0) 0x2779200 con 0x27b9600

  -298> 2013-10-22 11:55:31.104138 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2221013279 0 0) 0x2820a80 con 0x27b9600

  -297> 2013-10-22 11:55:31.104157 7f342c593700 10 monclient: handle_subscribe_ack sent 2013-10-22 11:55:31.102649 renew after 2013-10-22 11:58:01.102649

  -296> 2013-10-22 11:55:31.104182 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 10 ==== mdsmap(e 19375) v1 ==== 615+0+0 (2285621617 0 0) 0x27a1800 con 0x27b9600

  -295> 2013-10-22 11:55:31.104196 7f342c593700  5 mds.-1.0 handle_mds_map epoch 19375 from mon.2

  -294> 2013-10-22 11:55:31.104543 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 11 ==== osd_map(6647..6647 src has 6100..6647) v3 ==== 26502+0+0 (496991870 0 0) 0x2779f80 con 0x27b9600

  -293> 2013-10-22 11:55:31.104581 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2221013279 0 0) 0x28201c0 con 0x27b9600

  -292> 2013-10-22 11:55:31.104595 7f342c593700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring

  -291> 2013-10-22 11:55:31.550766 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 13 ==== mdsmap(e 19376) v1 ==== 607+0+0 (48834080 0 0) 0x27a1400 con 0x27b9600

  -290> 2013-10-22 11:55:31.550804 7f342c593700  5 mds.-1.-1 handle_mds_map epoch 19376 from mon.2

  -289> 2013-10-22 11:55:31.550858 7f342c593700  1 -- 192.168.6.101:6808/14609 mark_down 192.168.6.101:6808/14351 -- pipe dne

  -288> 2013-10-22 11:55:31.550871 7f342c593700  1 mds.-1.0 handle_mds_map standby

  -287> 2013-10-22 11:55:32.151584 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 14 ==== mdsmap(e 19377) v1 ==== 615+0+0 (3721236452 0 0) 0x27a1c00 con 0x27b9600

  -286> 2013-10-22 11:55:32.151622 7f342c593700  5 mds.-1.0 handle_mds_map epoch 19377 from mon.2

  -285> 2013-10-22 11:55:32.151652 7f342c593700  1 mds.0.19 handle_mds_map i am now mds.0.19

  -284> 2013-10-22 11:55:32.151658 7f342c593700  1 mds.0.19 handle_mds_map state change up:standby --> up:replay

  -283> 2013-10-22 11:55:32.151661 7f342c593700  1 mds.0.19 replay_start

  -282> 2013-10-22 11:55:32.151673 7f342c593700  1 mds.0.19  recovery set is 1

  -281> 2013-10-22 11:55:32.151675 7f342c593700  1 mds.0.19  need osdmap epoch 6648, have 6647

  -280> 2013-10-22 11:55:32.151677 7f342c593700  1 mds.0.19  waiting for osdmap 6648 (which blacklists prior instance)

  -279> 2013-10-22 11:55:32.151690 7f342c593700 10 monclient: renew_subs

  -278> 2013-10-22 11:55:32.151694 7f342c593700 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

  -277> 2013-10-22 11:55:32.151699 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- mon_subscribe({mdsmap=19378+,monmap=4+,osdmap=6648}) v2 -- ?+0 0x28201c0 con 0x27b9600

  -276> 2013-10-22 11:55:32.152354 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 15 ==== osd_map(6648..6648 src has 6100..6648) v3 ==== 312+0+0 (3225924099 0 0) 0x2779d40 con 0x27b9600

  -275> 2013-10-22 11:55:32.152429 7f342c593700  2 mds.0.19 boot_start 1: opening inotable

  -274> 2013-10-22 11:55:32.152543 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6802/17847 -- osd_op(mds.0.19:1 mds0_inotable [read 0~0] 1.b852b893 e6648) v4 -- ?+0 0x2779f80 con 0x27ba3c0

  -273> 2013-10-22 11:55:32.152586 7f342c593700  2 mds.0.19 boot_start 1: opening sessionmap

  -272> 2013-10-22 11:55:32.152673 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6804/21951 -- osd_op(mds.0.19:2 mds0_sessionmap [read 0~0] 1.3270c60b e6648) v4 -- ?+0 0x2779200 con 0x27ba260

  -271> 2013-10-22 11:55:32.152697 7f342c593700  2 mds.0.19 boot_start 1: opening anchor table

  -270> 2013-10-22 11:55:32.152756 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6805/23076 -- osd_op(mds.0.19:3 mds_anchortable [read 0~0] 1.a977f6a7 e6648) v4 -- ?+0 0x2779440 con 0x27ba680

  -269> 2013-10-22 11:55:32.152794 7f342c593700  2 mds.0.19 boot_start 1: opening snap table

  -268> 2013-10-22 11:55:32.152821 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6804/21951 -- osd_op(mds.0.19:4 mds_snaptable [read 0~0] 1.d90270ad e6648) v4 -- ?+0 0x277aac0 con 0x27ba260

  -267> 2013-10-22 11:55:32.152883 7f342c593700  2 mds.0.19 boot_start 1: opening mds log

  -266> 2013-10-22 11:55:32.152893 7f342c593700  5 mds.0.log open discovering log bounds

  -265> 2013-10-22 11:55:32.152962 7f342c593700  1 mds.0.journaler(ro) recover start

  -264> 2013-10-22 11:55:32.152968 7f342c593700  1 mds.0.journaler(ro) read_head

  -263> 2013-10-22 11:55:32.153065 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6804/26797 -- osd_op(mds.0.19:5 200.00000000 [read 0~0] 1.844f3494 e6648) v4 -- ?+0 0x277a880 con 0x27bad60

  -262> 2013-10-22 11:55:32.153115 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 16 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (2221013279 0 0) 0x2820c40 con 0x27b9600

  -261> 2013-10-22 11:55:32.153132 7f342c593700 10 monclient: handle_subscribe_ack sent 2013-10-22 11:55:32.151692 renew after 2013-10-22 11:58:02.151692

  -260> 2013-10-22 11:55:32.154083 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.104:6804/21951

  -259> 2013-10-22 11:55:32.154132 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.105:6805/23076

  -258> 2013-10-22 11:55:32.154392 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.105:6802/17847

  -257> 2013-10-22 11:55:32.154435 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.103:6804/26797

  -256> 2013-10-22 11:55:32.156138 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.34 192.168.6.103:6804/26797 1 ==== osd_op_reply(5 200.00000000 [read 0~84] _ondisk_ = 0) v4 ==== 111+0+84 (2168020559 0 3078552249) 0x27a1e00 con 0x27bad60

  -255> 2013-10-22 11:55:32.156203 7f342c593700  1 mds.0.journaler(ro) _finish_read_head loghead(trim 3338665984, expire 3339757125, write 3777838120).  probing for end of log (from 3777838120)...

  -254> 2013-10-22 11:55:32.156244 7f342c593700  1 mds.0.journaler(ro) probing for end of the log

  -253> 2013-10-22 11:55:32.156367 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:6 200.00000384 [stat] 1.d18de1e3 rwordered e6648) v4 -- ?+0 0x2779d40 con 0x27baaa0

  -252> 2013-10-22 11:55:32.156461 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6802/26930 -- osd_op(mds.0.19:7 200.00000385 [stat] 1.c2c5b01a rwordered e6648) v4 -- ?+0 0x277a640 con 0x27bb9c0

  -251> 2013-10-22 11:55:32.156503 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.4 192.168.6.104:6804/21951 1 ==== osd_op_reply(2 mds0_sessionmap [read 0~210] _ondisk_ = 0) v4 ==== 114+0+210 (3865533639 0 2938319697) 0x27a2200 con 0x27ba260

  -250> 2013-10-22 11:55:32.156586 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.17 192.168.6.105:6805/23076 1 ==== osd_op_reply(3 mds_anchortable [read 0~5784] _ondisk_ = 0) v4 ==== 114+0+5784 (3419867908 0 3259227584) 0x27a2000 con 0x27ba680

  -249> 2013-10-22 11:55:32.156845 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.4 192.168.6.104:6804/21951 2 ==== osd_op_reply(4 mds_snaptable [read 0~46] _ondisk_ = 0) v4 ==== 112+0+46 (1772553684 0 2603725198) 0x27a1e00 con 0x27ba260

  -248> 2013-10-22 11:55:32.157562 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.104:6805/5393

  -247> 2013-10-22 11:55:32.157722 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.102:6802/26930

  -246> 2013-10-22 11:55:32.158418 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 1 ==== osd_op_reply(6 200.00000384 [stat] _ondisk_ = 0) v4 ==== 111+0+16 (2071324260 0 4183633320) 0x27a2200 con 0x27baaa0

  -245> 2013-10-22 11:55:32.169284 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.8 192.168.6.105:6802/17847 1 ==== osd_op_reply(1 mds0_inotable [read 0~34] _ondisk_ = 0) v4 ==== 112+0+34 (1835735708 0 1910755494) 0x27a2000 con 0x27ba3c0

  -244> 2013-10-22 11:55:32.275334 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.25 192.168.6.102:6802/26930 1 ==== osd_op_reply(7 200.00000385 [stat] ack = -2 (No such file or directory)) v4 ==== 111+0+0 (3179590955 0 0) 0x27a2400 con 0x27bb9c0

  -243> 2013-10-22 11:55:32.275388 7f342c593700  1 mds.0.journaler(ro) _finish_probe_end write_pos = 3777856966 (header had 3777838120). recovered.

  -242> 2013-10-22 11:55:32.275397 7f342c593700  2 mds.0.19 boot_start 2: loading/discovering base inodes

  -241> 2013-10-22 11:55:32.275413 7f342c593700  0 mds.0.cache creating system inode with ino:100

  -240> 2013-10-22 11:55:32.275578 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6801/7366 -- osd_op(mds.0.19:8 100.00000000 [getxattr inode] 1.c5265ab3 e6648) v4 -- ?+0 0x277cc80 con 0x27bb700

  -239> 2013-10-22 11:55:32.275684 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6805/4378 -- osd_op(mds.0.19:9 100.00000000.inode [read 0~0] 1.85dde07f e6648) v4 -- ?+0 0x277ca40 con 0x27bb440

  -238> 2013-10-22 11:55:32.275720 7f342c593700  0 mds.0.cache creating system inode with ino:1

  -237> 2013-10-22 11:55:32.275797 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6804/32670 -- osd_op(mds.0.19:10 1.00000000 [getxattr inode] 1.6b2cdaff e6648) v4 -- ?+0 0x277c5c0 con 0x27bc360

  -236> 2013-10-22 11:55:32.275898 7f342c593700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6800/29108 -- osd_op(mds.0.19:11 1.00000000.inode [read 0~0] 1.232c0e14 e6648) v4 -- ?+0 0x277c380 con 0x27bc0a0

  -235> 2013-10-22 11:55:32.276830 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.102:6801/7366

  -234> 2013-10-22 11:55:32.276952 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.101:6804/32670

  -233> 2013-10-22 11:55:32.276998 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.103:6805/4378

  -232> 2013-10-22 11:55:32.277512 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.105:6800/29108

  -231> 2013-10-22 11:55:32.278115 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.35 192.168.6.103:6805/4378 1 ==== osd_op_reply(9 100.00000000.inode [read 0~432] _ondisk_ = 0) v4 ==== 117+0+432 (2135573121 0 1815528136) 0x27a2600 con 0x27bb440

  -230> 2013-10-22 11:55:32.278225 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.13 192.168.6.101:6804/32670 1 ==== osd_op_reply(10 1.00000000 [getxattr] _ondisk_ = -61 (No data available)) v4 ==== 109+0+0 (2288260501 0 0) 0x27a2800 con 0x27bc360

  -229> 2013-10-22 11:55:32.278772 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.9 192.168.6.105:6800/29108 1 ==== osd_op_reply(11 1.00000000.inode [read 0~432] _ondisk_ = 0) v4 ==== 115+0+432 (58846582 0 3638428205) 0x27a1e00 con 0x27bc0a0

  -228> 2013-10-22 11:55:32.313729 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.23 192.168.6.102:6801/7366 1 ==== osd_op_reply(8 100.00000000 [getxattr] _ondisk_ = -61 (No data available)) v4 ==== 111+0+0 (3391677547 0 0) 0x27a2200 con 0x27bb700

  -227> 2013-10-22 11:55:32.313792 7f342c593700  2 mds.0.19 boot_start 3: replaying mds log

  -226> 2013-10-22 11:55:32.313992 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:12 200.0000031c [read 1091141~3103163] 1.b2837949 e6648) v4 -- ?+0 0x277e0c0 con 0x27bcba0

  -225> 2013-10-22 11:55:32.314049 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6800/29108 -- osd_op(mds.0.19:13 200.0000031d [read 0~4194304] 1.efba932b e6648) v4 -- ?+0 0x277e540 con 0x27bc0a0

  -224> 2013-10-22 11:55:32.314138 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6802/5207 -- osd_op(mds.0.19:14 200.0000031e [read 0~4194304] 1.406def4b e6648) v4 -- ?+0 0x277e300 con 0x27bcd00

  -223> 2013-10-22 11:55:32.314234 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6800/18748 -- osd_op(mds.0.19:15 200.0000031f [read 0~4194304] 1.9f7c52d5 e6648) v4 -- ?+0 0x277e540 con 0x27bd120

  -222> 2013-10-22 11:55:32.314337 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6801/10834 -- osd_op(mds.0.19:16 200.00000320 [read 0~4194304] 1.bd374796 e6648) v4 -- ?+0 0x277c380 con 0x27bd800

  -221> 2013-10-22 11:55:32.314386 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:17 200.00000321 [read 0~4194304] 1.7af45276 e6648) v4 -- ?+0 0x277d580 con 0x27baaa0

  -220> 2013-10-22 11:55:32.314441 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6802/17847 -- osd_op(mds.0.19:18 200.00000322 [read 0~4194304] 1.f4bd6283 e6648) v4 -- ?+0 0x277ee40 con 0x27ba3c0

  -219> 2013-10-22 11:55:32.314590 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6804/32670 -- osd_op(mds.0.19:19 200.00000323 [read 0~4194304] 1.6b175349 e6648) v4 -- ?+0 0x277ec00 con 0x27bc360

  -218> 2013-10-22 11:55:32.314920 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:20 200.00000324 [read 0~4194304] 1.974641f4 e6648) v4 -- ?+0 0x277e9c0 con 0x27baaa0

  -217> 2013-10-22 11:55:32.315059 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:21 200.00000325 [read 0~4194304] 1.47fee53a e6648) v4 -- ?+0 0x277e780 con 0x27bcba0

  -216> 2013-10-22 11:55:32.315111 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6800/29108 -- osd_op(mds.0.19:22 200.00000326 [read 0~4194304] 1.d8bdf668 e6648) v4 -- ?+0 0x277f080 con 0x27bc0a0

  -215> 2013-10-22 11:55:32.315299 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.102:6803/13427

  -214> 2013-10-22 11:55:32.315431 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.101:6800/18748

  -213> 2013-10-22 11:55:32.315678 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.103:6802/5207

  -212> 2013-10-22 11:55:32.317523 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.105:6801/10834

  -211> 2013-10-22 11:55:32.324443 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.13 192.168.6.101:6804/32670 2 ==== osd_op_reply(19 200.00000323 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (223565465 0 1486794614) 0x27a2400 con 0x27bc360

  -210> 2013-10-22 11:55:32.326386 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.12 192.168.6.101:6800/18748 1 ==== osd_op_reply(15 200.0000031f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3703886151 0 1398888068) 0x27a2600 con 0x27bd120

  -209> 2013-10-22 11:55:32.384267 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.32 192.168.6.103:6802/5207 1 ==== osd_op_reply(14 200.0000031e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (571874231 0 1722029850) 0x27a2a00 con 0x27bcd00

  -208> 2013-10-22 11:55:32.411801 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 2 ==== osd_op_reply(17 200.00000321 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2562370519 0 3449834067) 0x27a1e00 con 0x27baaa0

  -207> 2013-10-22 11:55:32.473643 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 3 ==== osd_op_reply(20 200.00000324 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2923820705 0 2744575767) 0x27a2800 con 0x27baaa0

  -206> 2013-10-22 11:55:32.518524 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 1 ==== osd_op_reply(12 200.0000031c [read 1091141~3103163] _ondisk_ = 0) v4 ==== 111+0+3103163 (51313703 0 3149451823) 0x27a2200 con 0x27bcba0

  -205> 2013-10-22 11:55:32.558991 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 2 ==== osd_op_reply(21 200.00000325 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3529755968 0 120567193) 0x27a2a00 con 0x27bcba0

  -204> 2013-10-22 11:55:32.594075 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.9 192.168.6.105:6800/29108 2 ==== osd_op_reply(13 200.0000031d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1383192443 0 1105536051) 0x27a2e00 con 0x27bc0a0

  -203> 2013-10-22 11:55:32.596754 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.8 192.168.6.105:6802/17847 2 ==== osd_op_reply(18 200.00000322 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2628812979 0 223746529) 0x27a3200 con 0x27ba3c0

  -202> 2013-10-22 11:55:32.601929 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.16 192.168.6.105:6801/10834 1 ==== osd_op_reply(16 200.00000320 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4108282505 0 1400121577) 0x27a3400 con 0x27bd800

  -201> 2013-10-22 11:55:32.602046 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6802/17847 -- osd_op(mds.0.19:23 200.00000327 [read 0~4194304] 1.282f4b2a e6648) v4 -- ?+0 0x2779f80 con 0x27ba3c0

  -200> 2013-10-22 11:55:32.602171 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6801/16389 -- osd_op(mds.0.19:24 200.00000328 [read 0~4194304] 1.6c206a4d e6648) v4 -- ?+0 0x277ad00 con 0x27bd540

  -199> 2013-10-22 11:55:32.602299 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:25 200.00000329 [read 0~4194304] 1.2c0792a3 e6648) v4 -- ?+0 0x277d580 con 0x27be460

  -198> 2013-10-22 11:55:32.602350 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:26 200.0000032a [read 0~4194304] 1.f64cf227 e6648) v4 -- ?+0 0x5209440 con 0x27baaa0

  -197> 2013-10-22 11:55:32.602391 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6800/18748 -- osd_op(mds.0.19:27 200.0000032b [read 0~4194304] 1.7aa5ac98 e6648) v4 -- ?+0 0x5209200 con 0x27bd120

  -196> 2013-10-22 11:55:32.602440 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6801/10834 -- osd_op(mds.0.19:28 200.0000032c [read 0~4194304] 1.dc29fd0e e6648) v4 -- ?+0 0x5208fc0 con 0x27bd800

  -195> 2013-10-22 11:55:32.962043 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.103:6801/16389

  -194> 2013-10-22 11:55:32.962098 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.104:6803/8548

  -193> 2013-10-22 11:55:32.962122 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.12 192.168.6.101:6800/18748 2 ==== osd_op_reply(27 200.0000032b [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1543882210 0 3468702210) 0x27a2800 con 0x27bd120

  -192> 2013-10-22 11:55:32.962183 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.16 192.168.6.105:6801/10834 2 ==== osd_op_reply(28 200.0000032c [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1564986407 0 3269779871) 0x27a3600 con 0x27bd800

  -191> 2013-10-22 11:55:32.962222 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.33 192.168.6.103:6801/16389 1 ==== osd_op_reply(24 200.00000328 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2875445119 0 342360108) 0x27a2400 con 0x27bd540

  -190> 2013-10-22 11:55:32.962258 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 1 ==== osd_op_reply(25 200.00000329 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (200896998 0 1335715980) 0x27a3200 con 0x27be460

  -189> 2013-10-22 11:55:32.962289 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.8 192.168.6.105:6802/17847 3 ==== osd_op_reply(23 200.00000327 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (918029340 0 2872545289) 0x27a3000 con 0x27ba3c0

  -188> 2013-10-22 11:55:32.962324 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 4 ==== osd_op_reply(26 200.0000032a [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1337000268 0 1305407095) 0x27a2e00 con 0x27baaa0

  -187> 2013-10-22 11:55:32.962354 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.9 192.168.6.105:6800/29108 3 ==== osd_op_reply(22 200.00000326 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1992842295 0 2858795740) 0x27a2c00 con 0x27bc0a0

  -186> 2013-10-22 11:55:32.962433 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6801/10834 -- osd_op(mds.0.19:29 200.0000032d [read 0~4194304] 1.9971d826 e6648) v4 -- ?+0 0x5208fc0 con 0x27bd800

  -185> 2013-10-22 11:55:32.962461 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6805/23076 -- osd_op(mds.0.19:30 200.0000032e [read 0~4194304] 1.f616680 e6648) v4 -- ?+0 0x5208b40 con 0x27ba680

  -184> 2013-10-22 11:55:32.962485 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6802/26930 -- osd_op(mds.0.19:31 200.0000032f [read 0~4194304] 1.3aada709 e6648) v4 -- ?+0 0x5208900 con 0x27bb9c0

  -183> 2013-10-22 11:55:32.962507 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6800/29108 -- osd_op(mds.0.19:32 200.00000330 [read 0~4194304] 1.60730877 e6648) v4 -- ?+0 0x277e9c0 con 0x27bc0a0

  -182> 2013-10-22 11:55:32.962539 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:33 200.00000331 [read 0~4194304] 1.c4d91cd0 e6648) v4 -- ?+0 0x2779d40 con 0x27be460

  -181> 2013-10-22 11:55:33.033937 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 2 ==== osd_op_reply(33 200.00000331 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3807732877 0 59178064) 0x27a3e00 con 0x27be460

  -180> 2013-10-22 11:55:33.125022 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.25 192.168.6.102:6802/26930 2 ==== osd_op_reply(31 200.0000032f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2933954058 0 1057540628) 0x27a4200 con 0x27bb9c0

  -179> 2013-10-22 11:55:33.130996 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.17 192.168.6.105:6805/23076 2 ==== osd_op_reply(30 200.0000032e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1367668207 0 282596934) 0x27a3600 con 0x27ba680

  -178> 2013-10-22 11:55:33.190864 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.16 192.168.6.105:6801/10834 3 ==== osd_op_reply(29 200.0000032d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1965019968 0 2304034912) 0x27a3400 con 0x27bd800

  -177> 2013-10-22 11:55:33.190893 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.9 192.168.6.105:6800/29108 4 ==== osd_op_reply(32 200.00000330 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1710368171 0 2200102950) 0x27a3000 con 0x27bc0a0

  -176> 2013-10-22 11:55:33.203394 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6801/7366 -- osd_op(mds.0.19:34 200.00000332 [read 0~4194304] 1.d6a68963 e6648) v4 -- ?+0 0x277f080 con 0x27bb700

  -175> 2013-10-22 11:55:33.205145 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6805/7770 -- osd_op(mds.0.19:35 200.00000333 [read 0~4194304] 1.e7328793 e6648) v4 -- ?+0 0x277f740 con 0x27beb40

  -174> 2013-10-22 11:55:33.205174 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:36 200.00000334 [read 0~4194304] 1.e881d3ee e6648) v4 -- ?+0 0x277f500 con 0x27be460

  -173> 2013-10-22 11:55:33.205221 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6806/30674 -- osd_op(mds.0.19:37 200.00000335 [read 0~4194304] 1.4463e440 e6648) v4 -- ?+0 0x2779d40 con 0x27be880

  -172> 2013-10-22 11:55:33.205244 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6806/30674 -- osd_op(mds.0.19:38 200.00000336 [read 0~4194304] 1.3bfbb988 e6648) v4 -- ?+0 0x277d580 con 0x27be880

  -171> 2013-10-22 11:55:33.353541 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6804/32670 -- osd_op(mds.0.19:39 200.00000337 [read 0~4194304] 1.440f5323 e6648) v4 -- ?+0 0x52098c0 con 0x27bc360

  -170> 2013-10-22 11:55:33.354857 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.101:6805/7770

  -169> 2013-10-22 11:55:33.354878 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.102:6806/30674

  -168> 2013-10-22 11:55:33.354892 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.14 192.168.6.101:6805/7770 1 ==== osd_op_reply(35 200.00000333 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1466977052 0 369087845) 0x27a2e00 con 0x27beb40

  -167> 2013-10-22 11:55:33.354927 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.23 192.168.6.102:6801/7366 2 ==== osd_op_reply(34 200.00000332 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (506277853 0 2265570938) 0x27a3e00 con 0x27bb700

  -166> 2013-10-22 11:55:33.354944 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 3 ==== osd_op_reply(36 200.00000334 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2820740509 0 1104700794) 0x27a3c00 con 0x27be460

  -165> 2013-10-22 11:55:33.358373 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.13 192.168.6.101:6804/32670 3 ==== osd_op_reply(39 200.00000337 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (549158843 0 3182530112) 0x27a2000 con 0x27bc360

  -164> 2013-10-22 11:55:33.404132 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.22 192.168.6.102:6806/30674 1 ==== osd_op_reply(38 200.00000336 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2844403526 0 1754631890) 0x27a4800 con 0x27be880

  -163> 2013-10-22 11:55:33.423507 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:40 200.00000338 [read 0~4194304] 1.41b9e8a5 e6648) v4 -- ?+0 0x5208900 con 0x27bcba0

  -162> 2013-10-22 11:55:33.423543 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6806/30674 -- osd_op(mds.0.19:41 200.00000339 [read 0~4194304] 1.95c987c2 e6648) v4 -- ?+0 0x2779440 con 0x27be880

  -161> 2013-10-22 11:55:33.423565 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6805/4378 -- osd_op(mds.0.19:42 200.0000033a [read 0~4194304] 1.e6d89bc5 e6648) v4 -- ?+0 0x277af40 con 0x27bb440

  -160> 2013-10-22 11:55:33.423637 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6802/26930 -- osd_op(mds.0.19:43 200.0000033b [read 0~4194304] 1.2bb82309 e6648) v4 -- ?+0 0x277a640 con 0x27bb9c0

  -159> 2013-10-22 11:55:33.423655 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6806/30674 -- osd_op(mds.0.19:44 200.0000033c [read 0~4194304] 1.d36bdea5 e6648) v4 -- ?+0 0x277d580 con 0x27be880

  -158> 2013-10-22 11:55:33.598179 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6806/23050 -- osd_op(mds.0.19:45 200.0000033d [read 0~4194304] 1.f06e7758 e6648) v4 -- ?+0 0x520b840 con 0x27bf4e0

  -157> 2013-10-22 11:55:33.598641 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.35 192.168.6.103:6805/4378 2 ==== osd_op_reply(42 200.0000033a [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2049652023 0 312490337) 0x27a4c00 con 0x27bb440

  -156> 2013-10-22 11:55:33.598660 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.101:6806/23050

  -155> 2013-10-22 11:55:33.598670 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 3 ==== osd_op_reply(40 200.00000338 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (51038765 0 3905441934) 0x27a2600 con 0x27bcba0

  -154> 2013-10-22 11:55:33.598682 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.22 192.168.6.102:6806/30674 2 ==== osd_op_reply(41 200.00000339 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1345154503 0 450278675) 0x27a5000 con 0x27be880

  -153> 2013-10-22 11:55:33.598693 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.25 192.168.6.102:6802/26930 3 ==== osd_op_reply(43 200.0000033b [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2494026804 0 338127714) 0x27a4000 con 0x27bb9c0

  -152> 2013-10-22 11:55:33.603543 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.15 192.168.6.101:6806/23050 1 ==== osd_op_reply(45 200.0000033d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2220355217 0 218938629) 0x27a4200 con 0x27bf4e0

  -151> 2013-10-22 11:55:34.081089 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.22 192.168.6.102:6806/30674 3 ==== osd_op_reply(37 200.00000335 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (993845909 0 1725208565) 0x27a4e00 con 0x27be880

  -150> 2013-10-22 11:55:34.081218 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:46 200.0000033e [read 0~4194304] 1.825bb12d e6648) v4 -- ?+0 0x520b3c0 con 0x27bcba0

  -149> 2013-10-22 11:55:34.081279 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6804/32670 -- osd_op(mds.0.19:47 200.0000033f [read 0~4194304] 1.5400cf31 e6648) v4 -- ?+0 0x520b180 con 0x27bc360

  -148> 2013-10-22 11:55:34.081327 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6804/26797 -- osd_op(mds.0.19:48 200.00000340 [read 0~4194304] 1.d148534a e6648) v4 -- ?+0 0x520af40 con 0x27bad60

  -147> 2013-10-22 11:55:34.081373 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6800/18748 -- osd_op(mds.0.19:49 200.00000341 [read 0~4194304] 1.3c6335a0 e6648) v4 -- ?+0 0x520ad00 con 0x27bd120

  -146> 2013-10-22 11:55:34.081415 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6801/16389 -- osd_op(mds.0.19:50 200.00000342 [read 0~4194304] 1.ff6e0b3f e6648) v4 -- ?+0 0x520aac0 con 0x27bd540

  -145> 2013-10-22 11:55:34.081574 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6801/7366 -- osd_op(mds.0.19:51 200.00000343 [read 0~4194304] 1.29c5d2dd e6648) v4 -- ?+0 0x277d580 con 0x27bb700

  -144> 2013-10-22 11:55:34.090807 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.13 192.168.6.101:6804/32670 4 ==== osd_op_reply(47 200.0000033f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2446032608 0 640686444) 0x27a3000 con 0x27bc360

  -143> 2013-10-22 11:55:34.098288 7f342b591700 10 monclient: tick

  -142> 2013-10-22 11:55:34.098308 7f342b591700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-10-22 11:55:04.098306)

  -141> 2013-10-22 11:55:34.098331 7f342b591700 10 monclient: renew subs? (now: 2013-10-22 11:55:34.098331; renew after: 2013-10-22 11:58:02.151692) -- no

  -140> 2013-10-22 11:55:34.258545 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.12 192.168.6.101:6800/18748 3 ==== osd_op_reply(49 200.00000341 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (439193369 0 1760330846) 0x27a1e00 con 0x27bd120

  -139> 2013-10-22 11:55:34.258853 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.33 192.168.6.103:6801/16389 2 ==== osd_op_reply(50 200.00000342 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4169165455 0 345058866) 0x27a3e00 con 0x27bd540

  -138> 2013-10-22 11:55:34.259849 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.22 192.168.6.102:6806/30674 4 ==== osd_op_reply(44 200.0000033c [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2993115597 0 731857918) 0x27a4600 con 0x27be880

  -137> 2013-10-22 11:55:34.261198 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.34 192.168.6.103:6804/26797 2 ==== osd_op_reply(48 200.00000340 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1835007742 0 2152423683) 0x27a5600 con 0x27bad60

  -136> 2013-10-22 11:55:34.261229 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 4 ==== osd_op_reply(46 200.0000033e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3069522706 0 583235200) 0x27a4800 con 0x27bcba0

  -135> 2013-10-22 11:55:34.277488 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.23 192.168.6.102:6801/7366 3 ==== osd_op_reply(51 200.00000343 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (403392712 0 3109164339) 0x27a2c00 con 0x27bb700

  -134> 2013-10-22 11:55:34.319358 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:52 200.00000344 [read 0~4194304] 1.3ee6cfa5 e6648) v4 -- ?+0 0x520b3c0 con 0x27baaa0

  -133> 2013-10-22 11:55:34.319401 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6800/29108 -- osd_op(mds.0.19:53 200.00000345 [read 0~4194304] 1.9fe6b14 e6648) v4 -- ?+0 0x5208900 con 0x27bc0a0

  -132> 2013-10-22 11:55:34.319424 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6805/7770 -- osd_op(mds.0.19:54 200.00000346 [read 0~4194304] 1.7c83fced e6648) v4 -- ?+0 0x520a880 con 0x27beb40

  -131> 2013-10-22 11:55:34.319449 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6806/23050 -- osd_op(mds.0.19:55 200.00000347 [read 0~4194304] 1.eb879cb8 e6648) v4 -- ?+0 0x520a640 con 0x27bf4e0

  -130> 2013-10-22 11:55:34.319477 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:56 200.00000348 [read 0~4194304] 1.f2aa2596 e6648) v4 -- ?+0 0x520a400 con 0x27baaa0

  -129> 2013-10-22 11:55:34.326306 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.14 192.168.6.101:6805/7770 2 ==== osd_op_reply(54 200.00000346 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (548025061 0 1277348298) 0x27a2600 con 0x27beb40

  -128> 2013-10-22 11:55:34.488615 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.15 192.168.6.101:6806/23050 2 ==== osd_op_reply(55 200.00000347 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3807428301 0 4252032190) 0x27a4000 con 0x27bf4e0

  -127> 2013-10-22 11:55:34.488849 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 5 ==== osd_op_reply(52 200.00000344 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (480571814 0 3065351071) 0x27a2a00 con 0x27baaa0

  -126> 2013-10-22 11:55:34.488971 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.9 192.168.6.105:6800/29108 5 ==== osd_op_reply(53 200.00000345 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2313258300 0 186468992) 0x27a3200 con 0x27bc0a0

  -125> 2013-10-22 11:55:34.489026 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 6 ==== osd_op_reply(56 200.00000348 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3328076476 0 2146227354) 0x27a2200 con 0x27baaa0

  -124> 2013-10-22 11:55:34.493155 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:57 200.00000349 [read 0~4194304] 1.9c893a10 e6648) v4 -- ?+0 0x5209d40 con 0x27bcba0

  -123> 2013-10-22 11:55:34.569143 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6801/10834 -- osd_op(mds.0.19:58 200.0000034a [read 0~4194304] 1.7e06dd58 e6648) v4 -- ?+0 0x277c5c0 con 0x27bd800

  -122> 2013-10-22 11:55:34.569187 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6806/23050 -- osd_op(mds.0.19:59 200.0000034b [read 0~4194304] 1.bd0e599b e6648) v4 -- ?+0 0x520d7c0 con 0x27bf4e0

  -121> 2013-10-22 11:55:34.569210 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6801/7366 -- osd_op(mds.0.19:60 200.0000034c [read 0~4194304] 1.83391310 e6648) v4 -- ?+0 0x520d580 con 0x27bb700

  -120> 2013-10-22 11:55:34.569240 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:61 200.0000034d [read 0~4194304] 1.de9ac304 e6648) v4 -- ?+0 0x520a400 con 0x27be460

  -119> 2013-10-22 11:55:34.569265 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6801/16389 -- osd_op(mds.0.19:62 200.0000034e [read 0~4194304] 1.ef93c5ad e6648) v4 -- ?+0 0x520b3c0 con 0x27bd540

  -118> 2013-10-22 11:55:34.569293 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6805/7770 -- osd_op(mds.0.19:63 200.0000034f [read 0~4194304] 1.84f4fef4 e6648) v4 -- ?+0 0x520d340 con 0x27beb40

  -117> 2013-10-22 11:55:34.576032 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.15 192.168.6.101:6806/23050 3 ==== osd_op_reply(59 200.0000034b [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1612877852 0 3080747193) 0x27a5000 con 0x27bf4e0

  -116> 2013-10-22 11:55:34.746859 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.14 192.168.6.101:6805/7770 3 ==== osd_op_reply(63 200.0000034f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3103511353 0 2419541655) 0x27a4c00 con 0x27beb40

  -115> 2013-10-22 11:55:34.748013 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.23 192.168.6.102:6801/7366 4 ==== osd_op_reply(60 200.0000034c [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3960522778 0 1764000436) 0x27a5e00 con 0x27bb700

  -114> 2013-10-22 11:55:34.748239 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 5 ==== osd_op_reply(57 200.00000349 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (254659131 0 679485269) 0x27a2000 con 0x27bcba0

  -113> 2013-10-22 11:55:34.748288 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.33 192.168.6.103:6801/16389 3 ==== osd_op_reply(62 200.0000034e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1225948006 0 4146136388) 0x27a2e00 con 0x27bd540

  -112> 2013-10-22 11:55:34.748893 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 4 ==== osd_op_reply(61 200.0000034d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3606716465 0 905922456) 0x27a5800 con 0x27be460

  -111> 2013-10-22 11:55:34.766068 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.16 192.168.6.105:6801/10834 4 ==== osd_op_reply(58 200.0000034a [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (482252551 0 1790613585) 0x27a2c00 con 0x27bd800

  -110> 2013-10-22 11:55:34.813530 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6806/23050 -- osd_op(mds.0.19:64 200.00000350 [read 0~4194304] 1.a05b6db5 e6648) v4 -- ?+0 0x520a880 con 0x27bf4e0

  -109> 2013-10-22 11:55:34.818563 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.15 192.168.6.101:6806/23050 4 ==== osd_op_reply(64 200.00000350 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (864540651 0 1272037732) 0x27a2a00 con 0x27bf4e0

  -108> 2013-10-22 11:55:34.822705 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6804/32670 -- osd_op(mds.0.19:65 200.00000351 [read 0~4194304] 1.14920a28 e6648) v4 -- ?+0 0x520b840 con 0x27bc360

  -107> 2013-10-22 11:55:34.822731 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6806/23050 -- osd_op(mds.0.19:66 200.00000352 [read 0~4194304] 1.aaa6c5e8 e6648) v4 -- ?+0 0x520a400 con 0x27bf4e0

  -106> 2013-10-22 11:55:34.822746 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6805/23076 -- osd_op(mds.0.19:67 200.00000353 [read 0~4194304] 1.cb039ffb e6648) v4 -- ?+0 0x520cec0 con 0x27ba680

  -105> 2013-10-22 11:55:34.822763 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6804/26797 -- osd_op(mds.0.19:68 200.00000354 [read 0~4194304] 1.b4a0d2df e6648) v4 -- ?+0 0x520cc80 con 0x27bad60

  -104> 2013-10-22 11:55:34.822778 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:69 200.00000355 [read 0~4194304] 1.e069416b e6648) v4 -- ?+0 0x520ca40 con 0x27baaa0

  -103> 2013-10-22 11:55:34.997978 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.13 192.168.6.101:6804/32670 5 ==== osd_op_reply(65 200.00000351 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (582696979 0 1604097802) 0x27a3400 con 0x27bc360

  -102> 2013-10-22 11:55:34.998012 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.15 192.168.6.101:6806/23050 5 ==== osd_op_reply(66 200.00000352 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1112720539 0 1007677565) 0x27a4000 con 0x27bf4e0

  -101> 2013-10-22 11:55:34.998031 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.17 192.168.6.105:6805/23076 3 ==== osd_op_reply(67 200.00000353 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4115882267 0 86802956) 0x27a2800 con 0x27ba680

  -100> 2013-10-22 11:55:34.998050 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.34 192.168.6.103:6804/26797 3 ==== osd_op_reply(68 200.00000354 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3622581734 0 1109929494) 0x27a5400 con 0x27bad60

   -99> 2013-10-22 11:55:34.998068 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 7 ==== osd_op_reply(69 200.00000355 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3158426562 0 1734397663) 0x27a4c00 con 0x27baaa0

   -98> 2013-10-22 11:55:35.001991 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6806/23050 -- osd_op(mds.0.19:70 200.00000356 [read 0~4194304] 1.92215256 e6648) v4 -- ?+0 0x277f500 con 0x27bf4e0

   -97> 2013-10-22 11:55:35.006969 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.15 192.168.6.101:6806/23050 6 ==== osd_op_reply(70 200.00000356 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2230134717 0 1323389583) 0x27a2600 con 0x27bf4e0

   -96> 2013-10-22 11:55:35.069497 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6802/26930 -- osd_op(mds.0.19:71 200.00000357 [read 0~4194304] 1.206419c0 e6648) v4 -- ?+0 0x277cc80 con 0x27bb9c0

   -95> 2013-10-22 11:55:35.069530 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6804/21951 -- osd_op(mds.0.19:72 200.00000358 [read 0~4194304] 1.cc2fdd5f e6648) v4 -- ?+0 0x277cec0 con 0x27ba260

   -94> 2013-10-22 11:55:35.069557 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:73 200.00000359 [read 0~4194304] 1.f6368f3b e6648) v4 -- ?+0 0x520d7c0 con 0x27bcba0

   -93> 2013-10-22 11:55:35.069581 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6805/4378 -- osd_op(mds.0.19:74 200.0000035a [read 0~4194304] 1.eff2bbc5 e6648) v4 -- ?+0 0x520de80 con 0x27bb440

   -92> 2013-10-22 11:55:35.069604 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6804/26797 -- osd_op(mds.0.19:75 200.0000035b [read 0~4194304] 1.8496641f e6648) v4 -- ?+0 0x520dc40 con 0x27bad60

   -91> 2013-10-22 11:55:35.153079 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.35 192.168.6.103:6805/4378 3 ==== osd_op_reply(74 200.0000035a [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1510118057 0 1411674102) 0x27a4a00 con 0x27bb440

   -90> 2013-10-22 11:55:35.299509 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6804/32670 -- osd_op(mds.0.19:76 200.0000035c [read 0~4194304] 1.3705b2da e6648) v4 -- ?+0 0x277f500 con 0x27bc360

   -89> 2013-10-22 11:55:35.299601 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6802/1973 -- osd_op(mds.0.19:77 200.0000035d [read 0~4194304] 1.3962e329 e6648) v4 -- ?+0 0x520d7c0 con 0x27bfa60

   -88> 2013-10-22 11:55:35.299621 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6804/26797 -- osd_op(mds.0.19:78 200.0000035e [read 0~4194304] 1.ba515874 e6648) v4 -- ?+0 0x5209d40 con 0x27bad60

   -87> 2013-10-22 11:55:35.299635 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6802/17847 -- osd_op(mds.0.19:79 200.0000035f [read 0~4194304] 1.bca1fd06 e6648) v4 -- ?+0 0x520ba80 con 0x27ba3c0

   -86> 2013-10-22 11:55:35.299648 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6805/7770 -- osd_op(mds.0.19:80 200.00000360 [read 0~4194304] 1.2e75cbfa e6648) v4 -- ?+0 0x52098c0 con 0x27beb40

   -85> 2013-10-22 11:55:35.472374 7f342a38d700 10 monclient: _send_mon_message to mon.c at 192.168.6.103:6789/0

   -84> 2013-10-22 11:55:35.472401 7f342a38d700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6789/0 -- mdsbeacon(222230/a up:replay seq 2 v19377) v2 -- ?+0 0x2839080 con 0x27b9600

   -83> 2013-10-22 11:55:35.472565 7f342c593700  5 mds.0.19 ms_handle_connect on 192.168.6.104:6802/1973

   -82> 2013-10-22 11:55:35.472583 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.4 192.168.6.104:6804/21951 3 ==== osd_op_reply(72 200.00000358 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2750325223 0 623676034) 0x27a1c00 con 0x27ba260

   -81> 2013-10-22 11:55:35.472611 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.34 192.168.6.103:6804/26797 4 ==== osd_op_reply(75 200.0000035b [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2519693294 0 3820079043) 0x27a6a00 con 0x27bad60

   -80> 2013-10-22 11:55:35.473829 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.25 192.168.6.102:6802/26930 4 ==== osd_op_reply(71 200.00000357 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1799782339 0 417797054) 0x27a4e00 con 0x27bb9c0

   -79> 2013-10-22 11:55:35.473865 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.13 192.168.6.101:6804/32670 6 ==== osd_op_reply(76 200.0000035c [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4018327000 0 562514050) 0x27a3600 con 0x27bc360

   -78> 2013-10-22 11:55:35.473891 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6802/1973 -- osd_op(mds.0.19:81 200.00000361 [read 0~4194304] 1.ee8d1f59 e6648) v4 -- ?+0 0x520dc40 con 0x27bfa60

   -77> 2013-10-22 11:55:35.474566 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 6 ==== osd_op_reply(73 200.00000359 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3185636867 0 3520130312) 0x27a3c00 con 0x27bcba0

   -76> 2013-10-22 11:55:35.536066 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.14 192.168.6.101:6805/7770 4 ==== osd_op_reply(80 200.00000360 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3187182111 0 1821452086) 0x27a7000 con 0x27beb40

   -75> 2013-10-22 11:55:35.536117 7f342c593700  1 -- 192.168.6.101:6808/14609 <== mon.2 192.168.6.103:6789/0 17 ==== mdsbeacon(222230/a up:replay seq 2 v19377) v2 ==== 103+0+0 (1160586188 0 0) 0x2838dc0 con 0x27b9600

   -74> 2013-10-22 11:55:35.536134 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.2 192.168.6.104:6802/1973 1 ==== osd_op_reply(77 200.0000035d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (757974614 0 2318131477) 0x27a7200 con 0x27bfa60

   -73> 2013-10-22 11:55:35.536146 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.8 192.168.6.105:6802/17847 4 ==== osd_op_reply(79 200.0000035f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2499265364 0 420409409) 0x27a7800 con 0x27ba3c0

   -72> 2013-10-22 11:55:35.536158 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.34 192.168.6.103:6804/26797 5 ==== osd_op_reply(78 200.0000035e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4056071546 0 49720687) 0x27a6800 con 0x27bad60

   -71> 2013-10-22 11:55:35.536169 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.2 192.168.6.104:6802/1973 2 ==== osd_op_reply(81 200.00000361 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3919175214 0 4170173240) 0x27a2e00 con 0x27bfa60

   -70> 2013-10-22 11:55:35.536543 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:82 200.00000362 [read 0~4194304] 1.5143afe5 e6648) v4 -- ?+0 0x520ee40 con 0x27bcba0

   -69> 2013-10-22 11:55:35.536564 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6802/17847 -- osd_op(mds.0.19:83 200.00000363 [read 0~4194304] 1.51dcf9e5 e6648) v4 -- ?+0 0x520ec00 con 0x27ba3c0

   -68> 2013-10-22 11:55:35.536582 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6806/30674 -- osd_op(mds.0.19:84 200.00000364 [read 0~4194304] 1.58044d6d e6648) v4 -- ?+0 0x520e9c0 con 0x27be880

   -67> 2013-10-22 11:55:35.536604 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6804/21951 -- osd_op(mds.0.19:85 200.00000365 [read 0~4194304] 1.58f3a81 e6648) v4 -- ?+0 0x520a400 con 0x27ba260

   -66> 2013-10-22 11:55:35.536625 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:86 200.00000366 [read 0~4194304] 1.4576d0a0 e6648) v4 -- ?+0 0x277f740 con 0x27be460

   -65> 2013-10-22 11:55:35.613009 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 5 ==== osd_op_reply(86 200.00000366 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3737466672 0 263811339) 0x27a5200 con 0x27be460

   -64> 2013-10-22 11:55:35.774856 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6805/23076 -- osd_op(mds.0.19:87 200.00000367 [read 0~4194304] 1.f06d3353 e6648) v4 -- ?+0 0x8ad2ac0 con 0x27ba680

   -63> 2013-10-22 11:55:35.774899 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:88 200.00000368 [read 0~4194304] 1.d306f757 e6648) v4 -- ?+0 0x8ad2880 con 0x27be460

   -62> 2013-10-22 11:55:35.774921 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6800/29108 -- osd_op(mds.0.19:89 200.00000369 [read 0~4194304] 1.6d679ac3 e6648) v4 -- ?+0 0x8ad2640 con 0x27bc0a0

   -61> 2013-10-22 11:55:35.774947 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6801/16389 -- osd_op(mds.0.19:90 200.0000036a [read 0~4194304] 1.2563b3cf e6648) v4 -- ?+0 0x8ad2400 con 0x27bd540

   -60> 2013-10-22 11:55:35.774968 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6805/23076 -- osd_op(mds.0.19:91 200.0000036b [read 0~4194304] 1.7556a761 e6648) v4 -- ?+0 0x8ad21c0 con 0x27ba680

   -59> 2013-10-22 11:55:35.958072 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.4 192.168.6.104:6804/21951 4 ==== osd_op_reply(85 200.00000365 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1391053573 0 3304631731) 0x27a2a00 con 0x27ba260

   -58> 2013-10-22 11:55:35.958110 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.17 192.168.6.105:6805/23076 4 ==== osd_op_reply(87 200.00000367 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2624741587 0 3618476497) 0x27a7e00 con 0x27ba680

   -57> 2013-10-22 11:55:35.958126 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.9 192.168.6.105:6800/29108 6 ==== osd_op_reply(89 200.00000369 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (169808337 0 2061265654) 0x27a2400 con 0x27bc0a0

   -56> 2013-10-22 11:55:35.958142 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 7 ==== osd_op_reply(82 200.00000362 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2925506333 0 3261677863) 0x27a5400 con 0x27bcba0

   -55> 2013-10-22 11:55:35.958158 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.33 192.168.6.103:6801/16389 4 ==== osd_op_reply(90 200.0000036a [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1935835384 0 4223399646) 0x27a4a00 con 0x27bd540

   -54> 2013-10-22 11:55:35.958184 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6802/1973 -- osd_op(mds.0.19:92 200.0000036c [read 0~4194304] 1.dff1f99a e6648) v4 -- ?+0 0x8ad1d40 con 0x27bfa60

   -53> 2013-10-22 11:55:35.960153 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 6 ==== osd_op_reply(88 200.00000368 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (567474254 0 1635321155) 0x27a4600 con 0x27be460

   -52> 2013-10-22 11:55:36.027267 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.22 192.168.6.102:6806/30674 5 ==== osd_op_reply(84 200.00000364 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4284555363 0 2020622901) 0x27a4400 con 0x27be880

   -51> 2013-10-22 11:55:36.027306 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.2 192.168.6.104:6802/1973 3 ==== osd_op_reply(92 200.0000036c [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2832204058 0 3607505672) 0x27a2000 con 0x27bfa60

   -50> 2013-10-22 11:55:36.027319 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.8 192.168.6.105:6802/17847 5 ==== osd_op_reply(83 200.00000363 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (446379322 0 1646416750) 0x27a7600 con 0x27ba3c0

   -49> 2013-10-22 11:55:36.027338 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.17 192.168.6.105:6805/23076 5 ==== osd_op_reply(91 200.0000036b [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1840881987 0 3634405183) 0x27a7c00 con 0x27ba680

   -48> 2013-10-22 11:55:36.027365 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:93 200.0000036d [read 0~4194304] 1.9502ae6d e6648) v4 -- ?+0 0x8ad18c0 con 0x27bcba0

   -47> 2013-10-22 11:55:36.027392 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6801/7366 -- osd_op(mds.0.19:94 200.0000036e [read 0~4194304] 1.1d9ad902 e6648) v4 -- ?+0 0x8ad1680 con 0x27bb700

   -46> 2013-10-22 11:55:36.027412 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:95 200.0000036f [read 0~4194304] 1.3c16950f e6648) v4 -- ?+0 0x8ad1440 con 0x27bcba0

   -45> 2013-10-22 11:55:36.027457 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6805/23076 -- osd_op(mds.0.19:96 200.00000370 [read 0~4194304] 1.972dd27c e6648) v4 -- ?+0 0x520a400 con 0x27ba680

   -44> 2013-10-22 11:55:36.027476 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6802/1973 -- osd_op(mds.0.19:97 200.00000371 [read 0~4194304] 1.c5a5d7a2 e6648) v4 -- ?+0 0x277cec0 con 0x27bfa60

   -43> 2013-10-22 11:55:36.201641 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.2 192.168.6.104:6802/1973 4 ==== osd_op_reply(97 200.00000371 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1804215373 0 1385014679) 0x27a4e00 con 0x27bfa60

   -42> 2013-10-22 11:55:36.203368 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.17 192.168.6.105:6805/23076 6 ==== osd_op_reply(96 200.00000370 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (591456299 0 3885403369) 0x27a7a00 con 0x27ba680

   -41> 2013-10-22 11:55:36.271864 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6802/26930 -- osd_op(mds.0.19:98 200.00000372 [read 0~4194304] 1.4ad1c74 e6648) v4 -- ?+0 0x277ca40 con 0x27bb9c0

   -40> 2013-10-22 11:55:36.271904 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6804/26797 -- osd_op(mds.0.19:99 200.00000373 [read 0~4194304] 1.20173d10 e6648) v4 -- ?+0 0x277d340 con 0x27bad60

   -39> 2013-10-22 11:55:36.271926 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:100 200.00000374 [read 0~4194304] 1.b7eb8340 e6648) v4 -- ?+0 0x8ad0b40 con 0x27baaa0

   -38> 2013-10-22 11:55:36.271945 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6800/18748 -- osd_op(mds.0.19:101 200.00000375 [read 0~4194304] 1.dd1866bb e6648) v4 -- ?+0 0x8ad2640 con 0x27bd120

   -37> 2013-10-22 11:55:36.271964 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:102 200.00000376 [read 0~4194304] 1.191badc5 e6648) v4 -- ?+0 0x8ad2400 con 0x27bcba0

   -36> 2013-10-22 11:55:36.450743 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.23 192.168.6.102:6801/7366 5 ==== osd_op_reply(94 200.0000036e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2391879790 0 1748960158) 0x27a5c00 con 0x27bb700

   -35> 2013-10-22 11:55:36.450797 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 8 ==== osd_op_reply(95 200.0000036f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3597705126 0 56020184) 0x27a2800 con 0x27bcba0

   -34> 2013-10-22 11:55:36.450823 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.12 192.168.6.101:6800/18748 4 ==== osd_op_reply(101 200.00000375 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2017988740 0 1584893491) 0x27a7000 con 0x27bd120

   -33> 2013-10-22 11:55:36.450864 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.34 192.168.6.103:6804/26797 6 ==== osd_op_reply(99 200.00000373 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (491489119 0 2529378384) 0x27a6600 con 0x27bad60

   -32> 2013-10-22 11:55:36.450892 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 8 ==== osd_op_reply(100 200.00000374 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3035689023 0 1042422449) 0x27a5000 con 0x27baaa0

   -31> 2013-10-22 11:55:36.450913 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.25 192.168.6.102:6802/26930 5 ==== osd_op_reply(98 200.00000372 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1455092536 0 3564974882) 0x27a4200 con 0x27bb9c0

   -30> 2013-10-22 11:55:36.450931 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 9 ==== osd_op_reply(93 200.0000036d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2046830680 0 1745386691) 0x27a4000 con 0x27bcba0

   -29> 2013-10-22 11:55:36.450957 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 10 ==== osd_op_reply(102 200.00000376 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2280129007 0 2352950325) 0x27a3400 con 0x27bcba0

   -28> 2013-10-22 11:55:36.451002 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:103 200.00000377 [read 0~4194304] 1.9bd74fd5 e6648) v4 -- ?+0 0x8ad1d40 con 0x27bcba0

   -27> 2013-10-22 11:55:36.520882 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6801/7366 -- osd_op(mds.0.19:104 200.00000378 [read 0~4194304] 1.6ded1587 e6648) v4 -- ?+0 0x520cec0 con 0x27bb700

   -26> 2013-10-22 11:55:36.520921 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6804/21951 -- osd_op(mds.0.19:105 200.00000379 [read 0~4194304] 1.ed16b1b0 e6648) v4 -- ?+0 0x520e780 con 0x27ba260

   -25> 2013-10-22 11:55:36.520936 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.105:6802/17847 -- osd_op(mds.0.19:106 200.0000037a [read 0~4194304] 1.1416d55d e6648) v4 -- ?+0 0x520a400 con 0x27ba3c0

   -24> 2013-10-22 11:55:36.520951 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6803/8548 -- osd_op(mds.0.19:107 200.0000037b [read 0~4194304] 1.46affd2c e6648) v4 -- ?+0 0x520c380 con 0x27be460

   -23> 2013-10-22 11:55:36.520969 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6800/18748 -- osd_op(mds.0.19:108 200.0000037c [read 0~4194304] 1.8503387e e6648) v4 -- ?+0 0x520c140 con 0x27bd120

   -22> 2013-10-22 11:55:36.772278 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.12 192.168.6.101:6800/18748 5 ==== osd_op_reply(108 200.0000037c [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2907776114 0 57077318) 0x27a3c00 con 0x27bd120

   -21> 2013-10-22 11:55:36.772330 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.3 192.168.6.104:6803/8548 7 ==== osd_op_reply(107 200.0000037b [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3561528379 0 1932758060) 0x27a4e00 con 0x27be460

   -20> 2013-10-22 11:55:36.772356 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.8 192.168.6.105:6802/17847 6 ==== osd_op_reply(106 200.0000037a [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2147299793 0 3266050168) 0x27a7400 con 0x27ba3c0

   -19> 2013-10-22 11:55:36.772377 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.4 192.168.6.104:6804/21951 5 ==== osd_op_reply(105 200.00000379 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2531290040 0 2404314304) 0x27a2c00 con 0x27ba260

   -18> 2013-10-22 11:55:36.772405 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.23 192.168.6.102:6801/7366 6 ==== osd_op_reply(104 200.00000378 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4207836335 0 3140541078) 0x27a5a00 con 0x27bb700

   -17> 2013-10-22 11:55:36.772428 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 11 ==== osd_op_reply(103 200.00000377 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (4269712559 0 3606359958) 0x27a4400 con 0x27bcba0

   -16> 2013-10-22 11:55:36.772500 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6800/18748 -- osd_op(mds.0.19:109 200.0000037d [read 0~4194304] 1.1cf4e4d5 e6648) v4 -- ?+0 0x277cec0 con 0x27bd120

   -15> 2013-10-22 11:55:36.772547 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.101:6805/7770 -- osd_op(mds.0.19:110 200.0000037e [read 0~4194304] 1.a7977435 e6648) v4 -- ?+0 0x277d580 con 0x27beb40

   -14> 2013-10-22 11:55:36.772579 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6802/1973 -- osd_op(mds.0.19:111 200.0000037f [read 0~4194304] 1.954d0c e6648) v4 -- ?+0 0x277ad00 con 0x27bfa60

   -13> 2013-10-22 11:55:36.772603 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6804/21951 -- osd_op(mds.0.19:112 200.00000380 [read 0~4194304] 1.1ed0fa81 e6648) v4 -- ?+0 0x277c380 con 0x27ba260

   -12> 2013-10-22 11:55:36.772626 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.103:6805/4378 -- osd_op(mds.0.19:113 200.00000381 [read 0~4194304] 1.3a6f403f e6648) v4 -- ?+0 0x277f740 con 0x27bb440

   -11> 2013-10-22 11:55:36.779388 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.12 192.168.6.101:6800/18748 6 ==== osd_op_reply(109 200.0000037d [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2932010990 0 2743815302) 0x27a3600 con 0x27bd120

   -10> 2013-10-22 11:55:36.950864 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.14 192.168.6.101:6805/7770 5 ==== osd_op_reply(110 200.0000037e [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (2142130967 0 3370109071) 0x27a6e00 con 0x27beb40

    -9> 2013-10-22 11:55:36.951098 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6803/13427 -- osd_op(mds.0.19:114 200.00000382 [read 0~4194304] 1.25207dab e6648) v4 -- ?+0 0x8ad4380 con 0x27bcba0

    -8> 2013-10-22 11:55:37.015232 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.102:6802/26930 -- osd_op(mds.0.19:115 200.00000383 [read 0~4194304] 1.3547c1c0 e6648) v4 -- ?+0 0x8ad3f00 con 0x27bb9c0

    -7> 2013-10-22 11:55:37.015270 7f3428078700  1 -- 192.168.6.101:6808/14609 --> 192.168.6.104:6805/5393 -- osd_op(mds.0.19:116 200.00000384 [read 0~2983366] 1.d18de1e3 e6648) v4 -- ?+0 0x8ad3cc0 con 0x27baaa0

    -6> 2013-10-22 11:55:37.245401 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.2 192.168.6.104:6802/1973 5 ==== osd_op_reply(111 200.0000037f [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (47376710 0 2440776935) 0x27a6a00 con 0x27bfa60

    -5> 2013-10-22 11:55:37.245442 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.4 192.168.6.104:6804/21951 6 ==== osd_op_reply(112 200.00000380 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (1932117179 0 2911124566) 0x27a5800 con 0x27ba260

    -4> 2013-10-22 11:55:37.245455 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.5 192.168.6.104:6805/5393 9 ==== osd_op_reply(116 200.00000384 [read 0~2983366] _ondisk_ = 0) v4 ==== 111+0+2983366 (209065439 0 4001433004) 0x27a2200 con 0x27baaa0

    -3> 2013-10-22 11:55:37.245471 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.25 192.168.6.102:6802/26930 6 ==== osd_op_reply(115 200.00000383 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3689451975 0 98903803) 0x27a3800 con 0x27bb9c0

    -2> 2013-10-22 11:55:37.245483 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.35 192.168.6.103:6805/4378 4 ==== osd_op_reply(113 200.00000381 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (3388715537 0 1184149932) 0x27a6400 con 0x27bb440

    -1> 2013-10-22 11:55:37.245495 7f342c593700  1 -- 192.168.6.101:6808/14609 <== osd.24 192.168.6.102:6803/13427 12 ==== osd_op_reply(114 200.00000382 [read 0~4194304] _ondisk_ = 0) v4 ==== 111+0+4194304 (461021434 0 3928582164) 0x27a4600 con 0x27bcba0

     0> 2013-10-22 11:55:37.563382 7f3428078700 -1 mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f3428078700 time 2013-10-22 11:55:37.562600

mds/journal.cc: 1096: FAILED assert(in->first == p->dnfirst || (in->is_multiversion() && in->first > p->dnfirst))

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x399d) [0x65b0ad]

 2: (EUpdate::replay(MDS*)+0x3a) [0x663c0a]

 3: (MDLog::_replay_thread()+0x5cf) [0x82e17f]

 4: (MDLog::ReplayThread::entry()+0xd) [0x6393ad]

 5: (()+0x7d15) [0x7f3430fc2d15]

 6: (clone()+0x6d) [0x7f342fa3948d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---

   0/ 5 none

   0/ 1 lockdep

   0/ 1 context

   1/ 1 crush

   1/ 5 mds

   1/ 5 mds_balancer

   1/ 5 mds_locker

   1/ 5 mds_log

   1/ 5 mds_log_expire

   1/ 5 mds_migrator

   0/ 1 buffer

   0/ 1 timer

   0/ 1 filer

   0/ 1 striper

   0/ 1 objecter

   0/ 5 rados

   0/ 5 rbd

   0/ 5 journaler

   0/ 5 objectcacher

   0/ 5 client

   0/ 5 osd

   0/ 5 optracker

   0/ 5 objclass

   1/ 3 filestore

   1/ 3 journal

   0/ 5 ms

   1/ 5 mon

   0/10 monc

   1/ 5 paxos

   0/ 5 tp

   1/ 5 auth

   1/ 5 crypto

   1/ 1 finisher

   1/ 5 heartbeatmap

   1/ 5 perfcounter

   1/ 5 rgw

   1/ 5 hadoop

   1/ 5 javaclient

   1/ 5 asok

   1/ 1 throttle

  -2/-2 (syslog threshold)

  99/99 (stderr threshold)

  max_recent     10000

  max_new         1000

  log_file 

--- end dump of recent events ---

terminate called after throwing an instance of 'ceph::FailedAssertion'

*** Caught signal (Aborted) **

 in thread 7f3428078700

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 1: ceph-mds() [0xa0d4f2]

 2: (()+0xf000) [0x7f3430fca000]

 3: (gsignal()+0x35) [0x7f342f97cba5]

 4: (abort()+0x148) [0x7f342f97e358]

 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3430276e1d]

 6: (()+0x5eeb6) [0x7f3430274eb6]

 7: (()+0x5eee3) [0x7f3430274ee3]

 8: (()+0x5f10e) [0x7f343027510e]

 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x923b9a]

 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x399d) [0x65b0ad]

 11: (EUpdate::replay(MDS*)+0x3a) [0x663c0a]

 12: (MDLog::_replay_thread()+0x5cf) [0x82e17f]

 13: (MDLog::ReplayThread::entry()+0xd) [0x6393ad]

 14: (()+0x7d15) [0x7f3430fc2d15]

 15: (clone()+0x6d) [0x7f342fa3948d]

2013-10-22 11:55:37.572495 7f3428078700 -1 *** Caught signal (Aborted) **

 in thread 7f3428078700

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 1: ceph-mds() [0xa0d4f2]

 2: (()+0xf000) [0x7f3430fca000]

 3: (gsignal()+0x35) [0x7f342f97cba5]

 4: (abort()+0x148) [0x7f342f97e358]

 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3430276e1d]

 6: (()+0x5eeb6) [0x7f3430274eb6]

 7: (()+0x5eee3) [0x7f3430274ee3]

 8: (()+0x5f10e) [0x7f343027510e]

 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x923b9a]

 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x399d) [0x65b0ad]

 11: (EUpdate::replay(MDS*)+0x3a) [0x663c0a]

 12: (MDLog::_replay_thread()+0x5cf) [0x82e17f]

 13: (MDLog::ReplayThread::entry()+0xd) [0x6393ad]

 14: (()+0x7d15) [0x7f3430fc2d15]

 15: (clone()+0x6d) [0x7f342fa3948d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

     0> 2013-10-22 11:55:37.572495 7f3428078700 -1 *** Caught signal (Aborted) **

 in thread 7f3428078700

 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

 1: ceph-mds() [0xa0d4f2]

 2: (()+0xf000) [0x7f3430fca000]

 3: (gsignal()+0x35) [0x7f342f97cba5]

 4: (abort()+0x148) [0x7f342f97e358]

 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3430276e1d]

 6: (()+0x5eeb6) [0x7f3430274eb6]

 7: (()+0x5eee3) [0x7f3430274ee3]

 8: (()+0x5f10e) [0x7f343027510e]

 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x923b9a]

 10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x399d) [0x65b0ad]

 11: (EUpdate::replay(MDS*)+0x3a) [0x663c0a]

 12: (MDLog::_replay_thread()+0x5cf) [0x82e17f]

 13: (MDLog::ReplayThread::entry()+0xd) [0x6393ad]

 14: (()+0x7d15) [0x7f3430fc2d15]

 15: (clone()+0x6d) [0x7f342fa3948d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---

   0/ 5 none

   0/ 1 lockdep

   0/ 1 context

   1/ 1 crush

   1/ 5 mds

   1/ 5 mds_balancer

   1/ 5 mds_locker

   1/ 5 mds_log

   1/ 5 mds_log_expire

   1/ 5 mds_migrator

   0/ 1 buffer

   0/ 1 timer

   0/ 1 filer

   0/ 1 striper

   0/ 1 objecter

   0/ 5 rados

   0/ 5 rbd

   0/ 5 journaler

   0/ 5 objectcacher

   0/ 5 client

   0/ 5 osd

   0/ 5 optracker

   0/ 5 objclass

   1/ 3 filestore

   1/ 3 journal

   0/ 5 ms

   1/ 5 mon

   0/10 monc

   1/ 5 paxos

   0/ 5 tp

   1/ 5 auth

   1/ 5 crypto

   1/ 1 finisher

   1/ 5 heartbeatmap

   1/ 5 perfcounter

   1/ 5 rgw

   1/ 5 hadoop

   1/ 5 javaclient

   1/ 5 asok

   1/ 1 throttle

  -2/-2 (syslog threshold)

  99/99 (stderr threshold)

  max_recent     10000

  max_new         1000

  log_file 

--- end dump of recent events ---

Aborted

Regards,

Gagan

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com