An OSD always crash few minutes after start

[Chu Duc Minh <chu.ducminh@xxxxxxxxx>]

One of my OSDs have problems and can NOT be start. I tried to start many times but it always crash few minutes after start.

I think about two reasons to make it crash:

1. A read/write request to this OSD, but due to the corrupted volume/snapshot/parent-image/..., it crash.

2. The recovering process can NOT work properly due to the corrupted volumes/snapshot/parent-image/...

After many retry and check log, i guess the reason (2) is the main cause. Because  if (1) is the main cause, other OSDs (contain buggy volume/snapshot) will crash too.

State of my ceph cluster (just few seconds before crash time):

  111/57706299 objects degraded (0.001%)    
        14918 active+clean
                   1 active+clean+scrubbing+deep
                  52 active+recovery_wait+degraded
                   2 active+recovering+degraded

PS: i attach crash-dump log of that OSD in this email for your information.

Thank you!

  -635> 2014-11-07 16:14:56.841872 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting '_'
  -634> 2014-11-07 16:14:56.841885 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting 'snapset'
  -633> 2014-11-07 16:14:56.841890 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6
  -632> 2014-11-07 16:14:56.841922 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 = 0
  -631> 2014-11-07 16:14:56.841949 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting '_'
  -630> 2014-11-07 16:14:56.841961 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting 'snapset'
  -629> 2014-11-07 16:14:56.841966 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6
  -628> 2014-11-07 16:14:56.841990 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 = 0
  -627> 2014-11-07 16:14:56.841996 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x6aaa190 seq 5724748 r = 0, finisher 0x21a9d4a0 0
  -626> 2014-11-07 16:14:56.842000 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x6aaa190 seq 5724748 osr(6.2ff 0x5bbb130)/0x5bbb130
  -625> 2014-11-07 16:14:56.842044 7f8618bd3700  5 -- op tracker -- seq: 11695, time: 2014-11-07 16:14:56.842043, event: sub_op_applied, op: osd_sub_op(client.7867946.0:36305158 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149067 snapset=0=[]:[] snapc=0=[])
  -624> 2014-11-07 16:14:56.842067 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149067 (107767'13146066,107796'13149067] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] sub_op_modify_applied on 0x201971c0 op osd_sub_op(client.7867946.0:36305158 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149067 snapset=0=[]:[] snapc=0=[]) v11
  -623> 2014-11-07 16:14:56.842094 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149067 (107767'13146066,107796'13149067] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] op_applied version 107796'13149067
  -622> 2014-11-07 16:14:56.842154 7f8618bd3700  5 -- op tracker -- seq: 11695, time: 2014-11-07 16:14:56.842154, event: done, op: osd_sub_op(client.7867946.0:36305158 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149067 snapset=0=[]:[] snapc=0=[])
  -621> 2014-11-07 16:14:56.842374 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 1048576~524288/524288
  -620> 2014-11-07 16:14:56.842438 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 1572864~524288
  -619> 2014-11-07 16:14:56.849878 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 1572864~524288/524288
  -618> 2014-11-07 16:14:56.849947 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 2097152~524288
  -617> 2014-11-07 16:14:56.850304 7f85faef6700  1 -- 172.30.5.2:6803/25692 <== osd.9 172.30.5.1:6897/17062 164 ==== osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+33593 (1150990094 0 3715631076) 0x21938c00 con 0x1fe252c0
  -616> 2014-11-07 16:14:56.850339 7f85faef6700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.849985, event: header_read, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -615> 2014-11-07 16:14:56.850362 7f85faef6700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.849986, event: throttled, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -614> 2014-11-07 16:14:56.850378 7f85faef6700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.850289, event: all_read, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -613> 2014-11-07 16:14:56.850386 7f85faef6700  5 -- op tracker -- seq: 11696, time: 0.000000, event: dispatched, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -612> 2014-11-07 16:14:56.850394 7f85faef6700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
  -611> 2014-11-07 16:14:56.850426 7f85faef6700 20 osd.21 107796 should_share_map osd.9 172.30.5.1:6897/17062 107796
  -610> 2014-11-07 16:14:56.850434 7f85faef6700 15 osd.21 107796 enqueue_op 0x289f3500 prio 127 cost 33593 latency 0.000449 osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11
  -609> 2014-11-07 16:14:56.850480 7f8604e41700 10 osd.21 107796 dequeue_op 0x289f3500 prio 127 cost 33593 latency 0.000494 osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.2ff( v 107796'13149067 (107767'13146066,107796'13149067] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active]
  -608> 2014-11-07 16:14:56.850532 7f8604e41700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.850532, event: reached_pg, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -607> 2014-11-07 16:14:56.850547 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149067 (107767'13146066,107796'13149067] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] handle_message: 0x289f3500
  -606> 2014-11-07 16:14:56.850557 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 2097152~524288/524288
  -605> 2014-11-07 16:14:56.850563 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149067 (107767'13146066,107796'13149067] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] sub_op_modify trans bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 v 107796'13149068 (transaction) 176
  -604> 2014-11-07 16:14:56.850583 7f8604e41700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.850583, event: started, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -603> 2014-11-07 16:14:56.850611 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 2621440~524288
  -602> 2014-11-07 16:14:56.850602 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149067 (107767'13146066,107796'13149067] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] append_log log((107767'13146066,107796'13149067], crt=107796'13149065) [107796'13149068 (107796'13149067) modify   bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 by client.7867946.0:36305207 2014-11-07 16:14:56.848833]
  -601> 2014-11-07 16:14:56.850633 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146066,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] add_log_entry 107796'13149068 (107796'13149067) modify   bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 by client.7867946.0:36305207 2014-11-07 16:14:56.848833
  -600> 2014-11-07 16:14:56.850659 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146066,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] append_log  adding 1 keys
  -599> 2014-11-07 16:14:56.850681 7f8604e41700 10 trim log((107767'13146066,107796'13149068], crt=107796'13149065) to 107767'13146067
  -598> 2014-11-07 16:14:56.850734 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] append_log: trimming to 107796'13149065 entries 
  -597> 2014-11-07 16:14:56.850800 7f8604e41700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'13149068, trimmed: 107767'13146067
  -596> 2014-11-07 16:14:56.850843 7f8604e41700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.850843, event: started, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -595> 2014-11-07 16:14:56.850869 7f8604e41700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.2ff 0x5bbb130)/0x5bbb130
  -594> 2014-11-07 16:14:56.850879 7f8604e41700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724749 0x1f020478
  -593> 2014-11-07 16:14:56.850892 7f8604e41700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.850891, event: commit_queued_for_journal_write, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -592> 2014-11-07 16:14:56.850938 7f8604e41700 15 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] do_sub_op osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11
  -591> 2014-11-07 16:14:56.850971 7f8604e41700 10 osd.21 107796 dequeue_op 0x289f3500 finish
  -590> 2014-11-07 16:14:56.850984 7f861cbdb700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.850984, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -589> 2014-11-07 16:14:56.851312 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x62aa000 seq 5724749 osr(6.2ff 0x5bbb130) 0x1f020478
  -588> 2014-11-07 16:14:56.851340 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x62aa000 seq 5724749 osr(6.2ff 0x5bbb130) 35163 bytes   (queue has 1 ops and 35163 bytes)
  -587> 2014-11-07 16:14:56.851361 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x219fc1a0
  -586> 2014-11-07 16:14:56.851301 7f861c3da700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.851301, event: journaled_completion_queued, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -585> 2014-11-07 16:14:56.851378 7f86193d4700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x62aa000 seq 5724749 osr(6.2ff 0x5bbb130)/0x5bbb130 start
  -584> 2014-11-07 16:14:56.851391 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x1f020478
  -583> 2014-11-07 16:14:56.851401 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6af0b6/pglog_6.2ff/0//-1
  -582> 2014-11-07 16:14:56.851408 7f86183d2700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.851408, event: commit_sent, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -581> 2014-11-07 16:14:56.851440 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149066 active] sub_op_modify_commit on op osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.9
  -580> 2014-11-07 16:14:56.851467 7f86193d4700 10 filestore oid: 6af0b6/pglog_6.2ff/0//-1 not skipping op, *spos 5724749.0.0
  -579> 2014-11-07 16:14:56.851473 7f86193d4700 10 filestore  > header.spos 0.0.0
  -578> 2014-11-07 16:14:56.851479 7f86183d2700 20 osd.21 107796 share_map_peer 0x1fe252c0 already has epoch 107796
  -577> 2014-11-07 16:14:56.851484 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.1:6897/17062 -- osd_sub_op_reply(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] ondisk, result = 0) v2 -- ?+0 0x21859b80 con 0x1fe252c0
  -576> 2014-11-07 16:14:56.851521 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -575> 2014-11-07 16:14:56.851564 7f86193d4700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724749.0.1
  -574> 2014-11-07 16:14:56.851569 7f86193d4700 10 filestore  > header.spos 0.0.0
  -573> 2014-11-07 16:14:56.851615 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6af0b6/pglog_6.2ff/0//-1
  -572> 2014-11-07 16:14:56.851655 7f86193d4700 10 filestore oid: 6af0b6/pglog_6.2ff/0//-1 not skipping op, *spos 5724749.0.2
  -571> 2014-11-07 16:14:56.851659 7f86193d4700 10 filestore  > header.spos 0.0.0
  -570> 2014-11-07 16:14:56.851694 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6af0b6/pglog_6.2ff/0//-1
  -569> 2014-11-07 16:14:56.851735 7f86193d4700 10 filestore oid: 6af0b6/pglog_6.2ff/0//-1 not skipping op, *spos 5724749.0.3
  -568> 2014-11-07 16:14:56.851752 7f86193d4700 10 filestore  > header.spos 0.0.0
  -567> 2014-11-07 16:14:56.851917 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 object_size 4194304 write_size 4194304
  -566> 2014-11-07 16:14:56.851943 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
  -565> 2014-11-07 16:14:56.851948 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 object_size 4194304 write_size 4194304 = -95
  -564> 2014-11-07 16:14:56.851959 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 3547136~32768
  -563> 2014-11-07 16:14:56.852045 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 3547136~32768 = 32768
  -562> 2014-11-07 16:14:56.852090 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting '_'
  -561> 2014-11-07 16:14:56.852113 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting 'snapset'
  -560> 2014-11-07 16:14:56.852123 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6
  -559> 2014-11-07 16:14:56.852166 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 = 0
  -558> 2014-11-07 16:14:56.852210 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting '_'
  -557> 2014-11-07 16:14:56.852227 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting 'snapset'
  -556> 2014-11-07 16:14:56.852232 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6
  -555> 2014-11-07 16:14:56.852259 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 = 0
  -554> 2014-11-07 16:14:56.852270 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x62aa000 seq 5724749 r = 0, finisher 0x1fdc3f60 0
  -553> 2014-11-07 16:14:56.852275 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x62aa000 seq 5724749 osr(6.2ff 0x5bbb130)/0x5bbb130
  -552> 2014-11-07 16:14:56.852307 7f8618bd3700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.852307, event: sub_op_applied, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -551> 2014-11-07 16:14:56.852330 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] sub_op_modify_applied on 0x1f020380 op osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[]) v11
  -550> 2014-11-07 16:14:56.852357 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] op_applied version 107796'13149068
  -549> 2014-11-07 16:14:56.852391 7f8618bd3700  5 -- op tracker -- seq: 11696, time: 2014-11-07 16:14:56.852391, event: done, op: osd_sub_op(client.7867946.0:36305207 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149068 snapset=0=[]:[] snapc=0=[])
  -548> 2014-11-07 16:14:56.852747 7f85f75bd700  1 -- 172.30.5.2:6803/25692 <== osd.99 172.30.5.8:6829/26930 167 ==== osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+1337 (2400197422 0 1473251473) 0x21856400 con 0x1ff0fb00
  -547> 2014-11-07 16:14:56.852775 7f85f75bd700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.852569, event: header_read, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -546> 2014-11-07 16:14:56.852795 7f85f75bd700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.852571, event: throttled, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -545> 2014-11-07 16:14:56.852803 7f85f75bd700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.852734, event: all_read, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -544> 2014-11-07 16:14:56.852811 7f85f75bd700  5 -- op tracker -- seq: 11697, time: 0.000000, event: dispatched, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -543> 2014-11-07 16:14:56.852823 7f85f75bd700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
  -542> 2014-11-07 16:14:56.852836 7f85f75bd700 20 osd.21 107796 should_share_map osd.99 172.30.5.8:6829/26930 107796
  -541> 2014-11-07 16:14:56.852845 7f85f75bd700 15 osd.21 107796 enqueue_op 0x392afa00 prio 127 cost 1337 latency 0.000276 osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11
  -540> 2014-11-07 16:14:56.852891 7f8607646700 10 osd.21 107796 dequeue_op 0x392afa00 prio 127 cost 1337 latency 0.000321 osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.458( v 107796'18073331 lc 107791'18072999 (107712'18069992,107796'18073331] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8]
  -539> 2014-11-07 16:14:56.852939 7f8607646700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.852939, event: reached_pg, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -538> 2014-11-07 16:14:56.852951 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 2621440~524288/524288
  -537> 2014-11-07 16:14:56.852959 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073331 lc 107791'18072999 (107712'18069992,107796'18073331] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] handle_message: 0x392afa00
  -536> 2014-11-07 16:14:56.852974 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073331 lc 107791'18072999 (107712'18069992,107796'18073331] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] sub_op_modify trans a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 v 107796'18073332 (transaction) 176
  -535> 2014-11-07 16:14:56.852990 7f8607646700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.852989, event: started, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -534> 2014-11-07 16:14:56.853016 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 3145728~524288
  -533> 2014-11-07 16:14:56.853020 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073331 lc 107791'18072999 (107712'18069992,107796'18073331] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] append_log log((107712'18069992,107796'18073331], crt=107792'18073233) [107796'18073332 (107796'18073330) modify   a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 by client.7867946.0:36305223 2014-11-07 16:14:56.851000]
  -532> 2014-11-07 16:14:56.853054 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] add_log_entry 107796'18073332 (107796'18073330) modify   a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 by client.7867946.0:36305223 2014-11-07 16:14:56.851000
  -531> 2014-11-07 16:14:56.853116 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] append_log  adding 1 keys
  -530> 2014-11-07 16:14:56.853107 7f85faef6700  1 -- 172.30.5.2:6803/25692 <== osd.9 172.30.5.1:6897/17062 165 ==== osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+3897 (881022281 0 3603529510) 0x2227d800 con 0x1fe252c0
  -529> 2014-11-07 16:14:56.853140 7f85faef6700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.852959, event: header_read, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -528> 2014-11-07 16:14:56.853153 7f85faef6700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.852960, event: throttled, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -527> 2014-11-07 16:14:56.853139 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] append_log: trimming to 107791'18072999 entries 
  -526> 2014-11-07 16:14:56.853161 7f85faef6700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.853098, event: all_read, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -525> 2014-11-07 16:14:56.853169 7f85faef6700  5 -- op tracker -- seq: 11698, time: 0.000000, event: dispatched, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -524> 2014-11-07 16:14:56.853177 7f85faef6700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
  -523> 2014-11-07 16:14:56.853185 7f85faef6700 20 osd.21 107796 should_share_map osd.9 172.30.5.1:6897/17062 107796
  -522> 2014-11-07 16:14:56.853188 7f8607646700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'18073332, trimmed: 
  -521> 2014-11-07 16:14:56.853193 7f85faef6700 15 osd.21 107796 enqueue_op 0x222dec00 prio 127 cost 3897 latency 0.000233 osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11
  -520> 2014-11-07 16:14:56.853209 7f8607646700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.853209, event: started, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -519> 2014-11-07 16:14:56.853224 7f8607646700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.458 0x75766d0)/0x75766d0
  -518> 2014-11-07 16:14:56.853230 7f8607646700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724750 0x2837bef8
  -517> 2014-11-07 16:14:56.853237 7f8607646700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.853237, event: commit_queued_for_journal_write, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -516> 2014-11-07 16:14:56.853255 7f8604e41700 10 osd.21 107796 dequeue_op 0x222dec00 prio 127 cost 3897 latency 0.000295 osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active]
  -515> 2014-11-07 16:14:56.853271 7f8607646700 15 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] do_sub_op osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11
  -514> 2014-11-07 16:14:56.853310 7f8607646700 10 osd.21 107796 dequeue_op 0x392afa00 finish
  -513> 2014-11-07 16:14:56.853311 7f861cbdb700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.853310, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -512> 2014-11-07 16:14:56.853411 7f85f75bd700  1 -- 172.30.5.2:6803/25692 <== osd.99 172.30.5.8:6829/26930 168 ==== osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+4921 (3084594152 0 3586355403) 0xb2f5c00 con 0x1ff0fb00
  -511> 2014-11-07 16:14:56.853440 7f85f75bd700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853314, event: header_read, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -510> 2014-11-07 16:14:56.853470 7f85f75bd700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853315, event: throttled, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -509> 2014-11-07 16:14:56.853303 7f8604e41700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.853303, event: reached_pg, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -508> 2014-11-07 16:14:56.853487 7f85f75bd700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853401, event: all_read, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -507> 2014-11-07 16:14:56.853525 7f85f75bd700  5 -- op tracker -- seq: 11699, time: 0.000000, event: dispatched, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -506> 2014-11-07 16:14:56.853521 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] handle_message: 0x222dec00
  -505> 2014-11-07 16:14:56.853538 7f85f75bd700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
  -504> 2014-11-07 16:14:56.853547 7f85f75bd700 20 osd.21 107796 should_share_map osd.99 172.30.5.8:6829/26930 107796
  -503> 2014-11-07 16:14:56.853541 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] sub_op_modify trans bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 v 107796'13149069 (transaction) 176
  -502> 2014-11-07 16:14:56.853556 7f85f75bd700 15 osd.21 107796 enqueue_op 0x2122e900 prio 127 cost 4921 latency 0.000242 osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11
  -501> 2014-11-07 16:14:56.853558 7f8604e41700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.853558, event: started, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -500> 2014-11-07 16:14:56.853579 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149068 (107767'13146067,107796'13149068] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] append_log log((107767'13146067,107796'13149068], crt=107796'13149065) [107796'13149069 (107796'13149068) modify   bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 by client.7867946.0:36305226 2014-11-07 16:14:56.851845]
  -499> 2014-11-07 16:14:56.853592 7f8607646700 10 osd.21 107796 dequeue_op 0x2122e900 prio 127 cost 4921 latency 0.000277 osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8]
  -498> 2014-11-07 16:14:56.853620 7f8607646700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853620, event: reached_pg, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -497> 2014-11-07 16:14:56.853641 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] handle_message: 0x2122e900
  -496> 2014-11-07 16:14:56.853607 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146067,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] add_log_entry 107796'13149069 (107796'13149068) modify   bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 by client.7867946.0:36305226 2014-11-07 16:14:56.851845
  -495> 2014-11-07 16:14:56.853655 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] sub_op_modify trans a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 v 107796'18073333 (transaction) 176
  -494> 2014-11-07 16:14:56.853670 7f8607646700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853670, event: started, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -493> 2014-11-07 16:14:56.853669 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146067,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] append_log  adding 1 keys
  -492> 2014-11-07 16:14:56.853696 7f8604e41700 10 trim log((107767'13146067,107796'13149069], crt=107796'13149065) to 107767'13146068
  -491> 2014-11-07 16:14:56.853692 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073332 lc 107791'18072999 (107712'18069992,107796'18073332] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] append_log log((107712'18069992,107796'18073332], crt=107792'18073233) [107796'18073333 (107796'18073332) modify   a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 by client.7867946.0:36305224 2014-11-07 16:14:56.851065]
  -490> 2014-11-07 16:14:56.853724 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0xfcd5a50 seq 5724750 osr(6.458 0x75766d0) 0x2837bef8
  -489> 2014-11-07 16:14:56.853716 7f861c3da700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.853716, event: journaled_completion_queued, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -488> 2014-11-07 16:14:56.853732 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0xfcd5a50 seq 5724750 osr(6.458 0x75766d0) 2872 bytes   (queue has 1 ops and 2872 bytes)
  -487> 2014-11-07 16:14:56.853707 7f8604e41700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146068,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] append_log: trimming to 107796'13149065 entries 
  -486> 2014-11-07 16:14:56.853720 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] add_log_entry 107796'18073333 (107796'18073332) modify   a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 by client.7867946.0:36305224 2014-11-07 16:14:56.851065
  -485> 2014-11-07 16:14:56.853746 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x264bbf20
  -484> 2014-11-07 16:14:56.853754 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] append_log  adding 1 keys
  -483> 2014-11-07 16:14:56.853769 7f8619bd5700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0xfcd5a50 seq 5724750 osr(6.458 0x75766d0)/0x75766d0 start
  -482> 2014-11-07 16:14:56.853780 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x2837bef8
  -481> 2014-11-07 16:14:56.853770 7f8607646700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] append_log: trimming to 107791'18072999 entries 
  -480> 2014-11-07 16:14:56.853782 7f8604e41700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'13149069, trimmed: 107767'13146068
  -479> 2014-11-07 16:14:56.853789 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b865a/pglog_6.458/0//-1
  -478> 2014-11-07 16:14:56.853807 7f8607646700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'18073333, trimmed: 
  -477> 2014-11-07 16:14:56.853827 7f8619bd5700 10 filestore oid: 6b865a/pglog_6.458/0//-1 not skipping op, *spos 5724750.0.0
  -476> 2014-11-07 16:14:56.853831 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -475> 2014-11-07 16:14:56.853834 7f8607646700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853834, event: started, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -474> 2014-11-07 16:14:56.853828 7f8604e41700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.853828, event: started, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -473> 2014-11-07 16:14:56.853848 7f8607646700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.458 0x75766d0)/0x75766d0
  -472> 2014-11-07 16:14:56.853850 7f8604e41700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.2ff 0x5bbb130)/0x5bbb130
  -471> 2014-11-07 16:14:56.853854 7f8607646700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724751 0x2164e7f8
  -470> 2014-11-07 16:14:56.853862 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -469> 2014-11-07 16:14:56.853860 7f8607646700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853860, event: commit_queued_for_journal_write, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -468> 2014-11-07 16:14:56.853886 7f8619bd5700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724750.0.1
  -467> 2014-11-07 16:14:56.853890 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -466> 2014-11-07 16:14:56.853925 7f8604e41700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724752 0x1f0207f8
  -465> 2014-11-07 16:14:56.853925 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6b865a/pglog_6.458/0//-1
  -464> 2014-11-07 16:14:56.853916 7f8607646700 15 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] do_sub_op osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11
  -463> 2014-11-07 16:14:56.853935 7f8607646700 10 osd.21 107796 dequeue_op 0x2122e900 finish
  -462> 2014-11-07 16:14:56.853954 7f8619bd5700 10 filestore oid: 6b865a/pglog_6.458/0//-1 not skipping op, *spos 5724750.0.2
  -461> 2014-11-07 16:14:56.853958 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -460> 2014-11-07 16:14:56.853938 7f8604e41700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.853938, event: commit_queued_for_journal_write, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -459> 2014-11-07 16:14:56.853970 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b865a/pglog_6.458/0//-1
  -458> 2014-11-07 16:14:56.853954 7f861cbdb700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.853954, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -457> 2014-11-07 16:14:56.853954 7f86183d2700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.853954, event: commit_sent, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -456> 2014-11-07 16:14:56.853992 7f8619bd5700 10 filestore oid: 6b865a/pglog_6.458/0//-1 not skipping op, *spos 5724750.0.3
  -455> 2014-11-07 16:14:56.853995 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -454> 2014-11-07 16:14:56.853979 7f8604e41700 15 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146068,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] do_sub_op osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11
  -453> 2014-11-07 16:14:56.854001 7f8604e41700 10 osd.21 107796 dequeue_op 0x222dec00 finish
  -452> 2014-11-07 16:14:56.853995 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] sub_op_modify_commit on op osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.99
  -451> 2014-11-07 16:14:56.854020 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 object_size 4194304 write_size 4194304
  -450> 2014-11-07 16:14:56.854031 7f86183d2700 20 osd.21 107796 share_map_peer 0x1ff0fb00 already has epoch 107796
  -449> 2014-11-07 16:14:56.854013 7f861cbdb700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.854013, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -448> 2014-11-07 16:14:56.854036 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.8:6829/26930 -- osd_sub_op_reply(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] ondisk, result = 0) v2 -- ?+0 0x219c9840 con 0x1ff0fb00
  -447> 2014-11-07 16:14:56.854215 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
  -446> 2014-11-07 16:14:56.854223 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 object_size 4194304 write_size 4194304 = -95
  -445> 2014-11-07 16:14:56.854236 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 2908160~512
  -444> 2014-11-07 16:14:56.854386 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x20d189b0 seq 5724751 osr(6.458 0x75766d0) 0x2164e7f8
  -443> 2014-11-07 16:14:56.854377 7f861c3da700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.854376, event: journaled_completion_queued, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -442> 2014-11-07 16:14:56.854397 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x20d189b0 seq 5724751 osr(6.458 0x75766d0) 6456 bytes   (queue has 3 ops and 14795 bytes)
  -441> 2014-11-07 16:14:56.854402 7f861c3da700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.854402, event: journaled_completion_queued, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -440> 2014-11-07 16:14:56.854416 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x93d4600
  -439> 2014-11-07 16:14:56.854431 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x62aafa0 seq 5724752 osr(6.2ff 0x5bbb130) 0x1f0207f8
  -438> 2014-11-07 16:14:56.854438 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x62aafa0 seq 5724752 osr(6.2ff 0x5bbb130) 5467 bytes   (queue has 3 ops and 14795 bytes)
  -437> 2014-11-07 16:14:56.854444 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x328ba20
  -436> 2014-11-07 16:14:56.854465 7f86183d2700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.854465, event: commit_sent, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -435> 2014-11-07 16:14:56.854490 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] sub_op_modify_commit on op osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.99
  -434> 2014-11-07 16:14:56.854525 7f86183d2700 20 osd.21 107796 share_map_peer 0x1ff0fb00 already has epoch 107796
  -433> 2014-11-07 16:14:56.854530 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.8:6829/26930 -- osd_sub_op_reply(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] ondisk, result = 0) v2 -- ?+0 0x219c9dc0 con 0x1ff0fb00
  -432> 2014-11-07 16:14:56.854562 7f86183d2700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.854562, event: commit_sent, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -431> 2014-11-07 16:14:56.854585 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146068,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149067 active] sub_op_modify_commit on op osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.9
  -430> 2014-11-07 16:14:56.854627 7f86183d2700 20 osd.21 107796 share_map_peer 0x1fe252c0 already has epoch 107796
  -429> 2014-11-07 16:14:56.854632 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.1:6897/17062 -- osd_sub_op_reply(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] ondisk, result = 0) v2 -- ?+0 0x219c92c0 con 0x1fe252c0
  -428> 2014-11-07 16:14:56.856312 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 3145728~524288/524288
  -427> 2014-11-07 16:14:56.856400 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 3670016~524288
  -426> 2014-11-07 16:14:56.858880 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 3670016~524288/524288
  -425> 2014-11-07 16:14:56.858969 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 4194304~524288
  -424> 2014-11-07 16:14:56.858994 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) FileStore::read 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6 4194304~0/524288
  -423> 2014-11-07 16:14:56.859003 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) omap_get_header 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6
  -422> 2014-11-07 16:14:56.873309 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 2908160~512 = 512
  -421> 2014-11-07 16:14:56.873393 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting '_'
  -420> 2014-11-07 16:14:56.873419 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting 'snapset'
  -419> 2014-11-07 16:14:56.873429 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6
  -418> 2014-11-07 16:14:56.873480 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 = 0
  -417> 2014-11-07 16:14:56.873523 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting '_'
  -416> 2014-11-07 16:14:56.873546 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting 'snapset'
  -415> 2014-11-07 16:14:56.873556 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6
  -414> 2014-11-07 16:14:56.873595 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 = 0
  -413> 2014-11-07 16:14:56.873617 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0xfcd5a50 seq 5724750 r = 0, finisher 0x26385d00 0
  -412> 2014-11-07 16:14:56.873626 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0xfcd5a50 seq 5724750 osr(6.458 0x75766d0)/0x75766d0
  -411> 2014-11-07 16:14:56.873657 7f86193d4700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x20d189b0 seq 5724751 osr(6.458 0x75766d0)/0x75766d0 start
  -410> 2014-11-07 16:14:56.873667 7f8619bd5700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x62aafa0 seq 5724752 osr(6.2ff 0x5bbb130)/0x5bbb130 start
  -409> 2014-11-07 16:14:56.873674 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x2164e7f8
  -408> 2014-11-07 16:14:56.873675 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x1f0207f8
  -407> 2014-11-07 16:14:56.873686 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6af0b6/pglog_6.2ff/0//-1
  -406> 2014-11-07 16:14:56.873688 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b865a/pglog_6.458/0//-1
  -405> 2014-11-07 16:14:56.873687 7f8618bd3700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.873687, event: sub_op_applied, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -404> 2014-11-07 16:14:56.873713 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] sub_op_modify_applied on 0x2837be00 op osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[]) v11
  -403> 2014-11-07 16:14:56.873756 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] op_applied version 107796'18073332
  -402> 2014-11-07 16:14:56.873825 7f8618bd3700  5 -- op tracker -- seq: 11697, time: 2014-11-07 16:14:56.873824, event: done, op: osd_sub_op(client.7867946.0:36305223 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073332 snapset=0=[]:[] snapc=0=[])
  -401> 2014-11-07 16:14:56.880282 7f8619bd5700 10 filestore oid: 6af0b6/pglog_6.2ff/0//-1 not skipping op, *spos 5724752.0.0
  -400> 2014-11-07 16:14:56.880299 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -399> 2014-11-07 16:14:56.880310 7f86193d4700 10 filestore oid: 6b865a/pglog_6.458/0//-1 not skipping op, *spos 5724751.0.0
  -398> 2014-11-07 16:14:56.880317 7f86193d4700 10 filestore  > header.spos 0.0.0
  -397> 2014-11-07 16:14:56.880341 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -396> 2014-11-07 16:14:56.880368 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) get_omap_iterator 6.35a_head/29f6135a/rbd_data.d7a1d7f53e9.00000000000286fe/head//6
  -395> 2014-11-07 16:14:56.880375 7f8619bd5700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724752.0.1
  -394> 2014-11-07 16:14:56.880379 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -393> 2014-11-07 16:14:56.880380 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -392> 2014-11-07 16:14:56.880417 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6af0b6/pglog_6.2ff/0//-1
  -391> 2014-11-07 16:14:56.880432 7f86193d4700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724751.0.1
  -390> 2014-11-07 16:14:56.880437 7f86193d4700 10 filestore  > header.spos 0.0.0
  -389> 2014-11-07 16:14:56.880468 7f8619bd5700 10 filestore oid: 6af0b6/pglog_6.2ff/0//-1 not skipping op, *spos 5724752.0.2
  -388> 2014-11-07 16:14:56.880478 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -387> 2014-11-07 16:14:56.880518 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6af0b6/pglog_6.2ff/0//-1
  -386> 2014-11-07 16:14:56.880539 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) omap_get_values meta/a468ec03/snapmapper/0//-1
  -385> 2014-11-07 16:14:56.880546 7f8619bd5700 10 filestore oid: 6af0b6/pglog_6.2ff/0//-1 not skipping op, *spos 5724752.0.3
  -384> 2014-11-07 16:14:56.880572 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -383> 2014-11-07 16:14:56.880613 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 object_size 4194304 write_size 4194304
  -382> 2014-11-07 16:14:56.880630 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
  -381> 2014-11-07 16:14:56.880625 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6b865a/pglog_6.458/0//-1
  -380> 2014-11-07 16:14:56.880633 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 object_size 4194304 write_size 4194304 = -95
  -379> 2014-11-07 16:14:56.880639 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 1679360~3072
  -378> 2014-11-07 16:14:56.880668 7f8602e3d700 20 snap_mapper.get_snaps 9726135a/rbd_data.3d3f9c2b321d1b.0000000000023134/1986//6 1986
  -377> 2014-11-07 16:14:56.880679 7f86193d4700 10 filestore oid: 6b865a/pglog_6.458/0//-1 not skipping op, *spos 5724751.0.2
  -376> 2014-11-07 16:14:56.880679 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 1679360~3072 = 3072
  -375> 2014-11-07 16:14:56.880686 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) collection_getattrs /var/lib/ceph/osd/cloud-21/current/6.35a_head
  -374> 2014-11-07 16:14:56.880688 7f86193d4700 10 filestore  > header.spos 0.0.0
  -373> 2014-11-07 16:14:56.880711 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting '_'
  -372> 2014-11-07 16:14:56.880712 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b865a/pglog_6.458/0//-1
  -371> 2014-11-07 16:14:56.880721 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting 'snapset'
  -370> 2014-11-07 16:14:56.880726 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6
  -369> 2014-11-07 16:14:56.880726 7f8602e3d700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 257 getting 'info'
  -368> 2014-11-07 16:14:56.880743 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) collection_getattrs /var/lib/ceph/osd/cloud-21/current/6.35a_head = 0
  -367> 2014-11-07 16:14:56.880754 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 = 0
  -366> 2014-11-07 16:14:56.880760 7f86193d4700 10 filestore oid: 6b865a/pglog_6.458/0//-1 not skipping op, *spos 5724751.0.3
  -365> 2014-11-07 16:14:56.880764 7f86193d4700 10 filestore  > header.spos 0.0.0
  -364> 2014-11-07 16:14:56.880776 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting '_'
  -363> 2014-11-07 16:14:56.880784 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 648 getting 'snapset'
  -362> 2014-11-07 16:14:56.880747 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] build_scrub_map_chunk done.
  -361> 2014-11-07 16:14:56.880789 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6
  -360> 2014-11-07 16:14:56.880808 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 object_size 4194304 write_size 4194304
  -359> 2014-11-07 16:14:56.880818 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.2ff_head/bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 = 0
  -358> 2014-11-07 16:14:56.880824 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x62aafa0 seq 5724752 r = 0, finisher 0x20755800 0
  -357> 2014-11-07 16:14:56.880825 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
  -356> 2014-11-07 16:14:56.880829 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x62aafa0 seq 5724752 osr(6.2ff 0x5bbb130)/0x5bbb130
  -355> 2014-11-07 16:14:56.880830 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 object_size 4194304 write_size 4194304 = -95
  -354> 2014-11-07 16:14:56.880842 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 2912256~4096
  -353> 2014-11-07 16:14:56.880835 7f8602e3d700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrub state WAIT_REPLICAS
  -352> 2014-11-07 16:14:56.880851 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] wait for replicas to build scrub map
  -351> 2014-11-07 16:14:56.880877 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] snap_trimmer entry
  -350> 2014-11-07 16:14:56.880875 7f8618bd3700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.880875, event: sub_op_applied, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -349> 2014-11-07 16:14:56.880901 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 2912256~4096 = 4096
  -348> 2014-11-07 16:14:56.880899 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] snap_trimmer posting
  -347> 2014-11-07 16:14:56.880919 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] SnapTrimmer state<NotTrimming>: NotTrimming react
  -346> 2014-11-07 16:14:56.880900 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146068,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149068 active] sub_op_modify_applied on 0x1f020700 op osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[]) v11
  -345> 2014-11-07 16:14:56.880944 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting '_'
  -344> 2014-11-07 16:14:56.880891 7f8606e45700 10 osd.21 107796 dequeue_op 0x20da0e00 prio 63 cost 1937408 latency 3.268192 osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4 pg pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]
  -343> 2014-11-07 16:14:56.880940 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] SnapTrimmer state<NotTrimming>: NotTrimming: trimming 1907
  -342> 2014-11-07 16:14:56.880961 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting 'snapset'
  -341> 2014-11-07 16:14:56.880947 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.2ff( v 107796'13149069 (107767'13146068,107796'13149069] local-les=107794 n=9075 ec=164 les/c 107794/107796 107793/107793/104829) [9,21,88] r=1 lpr=107793 pi=106418-107792/39 luod=0'0 crt=107796'13149065 lcod 107796'13149068 active] op_applied version 107796'13149069
  -340> 2014-11-07 16:14:56.880961 7f8602e3d700 20 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] exit NotTrimming
  -339> 2014-11-07 16:14:56.880971 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6
  -338> 2014-11-07 16:14:56.880957 7f8606e45700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.880957, event: reached_pg, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -337> 2014-11-07 16:14:56.880987 7f8602e3d700 20 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] enter Trimming/TrimmingObjects
  -336> 2014-11-07 16:14:56.880986 7f8606e45700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] op_has_sufficient_caps pool=6 (volumes ) owner=0 need_read_cap=1 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes
  -335> 2014-11-07 16:14:56.880997 7f8618bd3700  5 -- op tracker -- seq: 11698, time: 2014-11-07 16:14:56.880996, event: done, op: osd_sub_op(client.7867946.0:36305226 6.2ff bce0a2ff/rbd_data.4a91062ae8944a.0000000000000c1f/head//6 [] v 107796'13149069 snapset=0=[]:[] snapc=0=[])
  -334> 2014-11-07 16:14:56.881032 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 = 0
  -333> 2014-11-07 16:14:56.881017 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] SnapTrimmer state<Trimming/TrimmingObjects>: TrimmingObjects react
  -332> 2014-11-07 16:14:56.881022 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] handle_message: 0x20da0e00
  -331> 2014-11-07 16:14:56.881072 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting '_'
  -330> 2014-11-07 16:14:56.881050 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] SnapTrimmer state<Trimming/TrimmingObjects>: TrimmingObjects: trimming snap 1907
  -329> 2014-11-07 16:14:56.881048 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_op osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4 may_write may_read -> write-ordered flags ack+ondisk+write
  -328> 2014-11-07 16:14:56.881082 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 608 getting 'snapset'
  -327> 2014-11-07 16:14:56.881084 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) get_omap_iterator meta/a468ec03/snapmapper/0//-1
  -326> 2014-11-07 16:14:56.881087 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6
  -325> 2014-11-07 16:14:56.881094 7f8606e45700 15 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 '_'
  -324> 2014-11-07 16:14:56.881114 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.458_head/a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 = 0
  -323> 2014-11-07 16:14:56.881122 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x20d189b0 seq 5724751 r = 0, finisher 0x264b9940 0
  -322> 2014-11-07 16:14:56.881126 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x20d189b0 seq 5724751 osr(6.458 0x75766d0)/0x75766d0
  -321> 2014-11-07 16:14:56.881167 7f8618bd3700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.881167, event: sub_op_applied, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -320> 2014-11-07 16:14:56.881209 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] sub_op_modify_applied on 0x2164e700 op osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[]) v11
  -319> 2014-11-07 16:14:56.881241 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.458( v 107796'18073333 lc 107791'18072999 (107712'18069992,107796'18073333] local-les=107796 n=9355 ec=164 les/c 107796/107791 107793/107793/105262) [99,62,21] r=2 lpr=107793 pi=106418-107792/41 luod=0'0 crt=107792'18073233 active m=8] op_applied version 107796'18073333
  -318> 2014-11-07 16:14:56.881259 7f8606e45700 10 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 '_' = 267
  -317> 2014-11-07 16:14:56.881275 7f8618bd3700  5 -- op tracker -- seq: 11699, time: 2014-11-07 16:14:56.881275, event: done, op: osd_sub_op(client.7867946.0:36305224 6.458 a20f8458/rbd_data.4a91062ae8944a.0000000000001b10/head//6 [] v 107796'18073333 snapset=0=[]:[] snapc=0=[])
  -316> 2014-11-07 16:14:56.881311 7f8606e45700 15 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 'snapset'
  -315> 2014-11-07 16:14:56.881333 7f8606e45700 10 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 'snapset' = 31
  -314> 2014-11-07 16:14:56.881331 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] SnapTrimmer state<Trimming/TrimmingObjects>: TrimmingObjects react trimming 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6
  -313> 2014-11-07 16:14:56.881364 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.9d8_head/805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6 '_'
  -312> 2014-11-07 16:14:56.881352 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] populate_obc_watchers bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6
  -311> 2014-11-07 16:14:56.881382 7f8606e45700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] ReplicatedPG::check_blacklisted_obc_watchers for obc bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6
  -310> 2014-11-07 16:14:56.881403 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] get_object_context: creating obc from disk: 0x22591b00
  -309> 2014-11-07 16:14:56.881423 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] get_object_context: 0x22591b00 bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 rwstate(none n=0 w=0) oi: bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6(107796'15115561 client.7912580.0:24681440 wrlock_by=unknown.0.0:0 dirty s 4194304 uv15115561) ssc: 0x2145f540 snapset: 0=[]:[]+head
  -308> 2014-11-07 16:14:56.881462 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] find_object_context bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 @head oi=bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6(107796'15115561 client.7912580.0:24681440 wrlock_by=unknown.0.0:0 dirty s 4194304 uv15115561)
  -307> 2014-11-07 16:14:56.881504 7f8606e45700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.881504, event: started, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -306> 2014-11-07 16:14:56.881585 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] execute_ctx 0x1040f200
  -305> 2014-11-07 16:14:56.881615 7f8606e45700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.881614, event: started, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -304> 2014-11-07 16:14:56.881663 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_op bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] ov 107796'15115561 av 107796'15115562 snapc 0=[] snapset 0=[]:[]+head
  -303> 2014-11-07 16:14:56.881692 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  taking ondisk_read_lock
  -302> 2014-11-07 16:14:56.881710 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_osd_op bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408]
  -301> 2014-11-07 16:14:56.881731 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_osd_op  stat
  -300> 2014-11-07 16:14:56.881752 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] stat oi has 4194304 2014-11-07 16:14:47.774951
  -299> 2014-11-07 16:14:56.881779 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_osd_op  set-alloc-hint object_size 4194304 write_size 4194304
  -298> 2014-11-07 16:14:56.881806 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_osd_op  write 2256896~1937408
  -297> 2014-11-07 16:14:56.881843 7f8606e45700 15 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_osd_op_effects on session 0x200bc500
  -296> 2014-11-07 16:14:56.881863 7f8606e45700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] make_writeable bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 snapset=0x2145f578  snapc=0=[]
  -295> 2014-11-07 16:14:56.881886 7f8606e45700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] make_writeable bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 done, snapset=0=[]:[]+head
  -294> 2014-11-07 16:14:56.881909 7f8606e45700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] finish_ctx bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 0x1040f200 op modify  
  -293> 2014-11-07 16:14:56.881964 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  set mtime to 2014-11-07 16:14:53.599891
  -292> 2014-11-07 16:14:56.882007 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  final snapset 0=[]:[]+head in bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6
  -291> 2014-11-07 16:14:56.882039 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  dropping ondisk_read_lock
  -290> 2014-11-07 16:14:56.882063 7f8606e45700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  zeroing write result code 0
  -289> 2014-11-07 16:14:56.882081 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] calc_trim_to 107712'15112560 -> 107712'15112561
  -288> 2014-11-07 16:14:56.882101 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] new_repop rep_tid 1754 on osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4
  -287> 2014-11-07 16:14:56.882161 7f8606e45700  7 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] issue_repop rep_tid 1754 o bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6
  -286> 2014-11-07 16:14:56.882200 7f8606e45700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.882200, event: waiting for subops from 61,77, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -285> 2014-11-07 16:14:56.882268 7f8606e45700 20 osd.21 107796 share_map_peer 0x1fe34b00 already has epoch 107796
  -284> 2014-11-07 16:14:56.882282 7f8606e45700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.5:6831/12481 -- osd_sub_op(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] v 107796'15115562 snapset=0=[]:[] snapc=0=[]) v11 -- ?+1938233 0x270d8c00 con 0x1fe34b00
  -283> 2014-11-07 16:14:56.882388 7f8606e45700 20 osd.21 107796 share_map_peer 0x1fe01580 already has epoch 107796
  -282> 2014-11-07 16:14:56.882395 7f8606e45700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.6:6841/32590 -- osd_sub_op(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] v 107796'15115562 snapset=0=[]:[] snapc=0=[]) v11 -- ?+1938233 0x28fdf800 con 0x1fe01580
  -281> 2014-11-07 16:14:56.882442 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115561 (107712'15112560,107796'15115561] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] append_log log((107712'15112560,107796'15115561], crt=107796'15115554) [107796'15115562 (107796'15115561) modify   bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 by client.7912580.0:24681633 2014-11-07 16:14:53.599891]
  -280> 2014-11-07 16:14:56.883359 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112560,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115554 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] add_log_entry 107796'15115562 (107796'15115561) modify   bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 by client.7912580.0:24681633 2014-11-07 16:14:53.599891
  -279> 2014-11-07 16:14:56.883400 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112560,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] append_log  adding 1 keys
  -278> 2014-11-07 16:14:56.883417 7f8606e45700 10 trim log((107712'15112560,107796'15115562], crt=107796'15115560) to 107712'15112561
  -277> 2014-11-07 16:14:56.883426 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] append_log: trimming to 107796'15115560 entries 107796'15115555 (107796'15115554) modify   bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 by client.7912580.0:24680233 2014-11-07 16:14:19.346205,107796'15115556 (107796'15115555) modify   bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 by client.7912580.0:24680802 2014-11-07 16:14:31.335911,107796'15115557 (107796'15115538) modify   8a10a35a/rbd_data.ad62ae2ae8944a.0000000000004998/head//6 by client.11364924.0:113843 2014-11-07 16:14:30.267596,107796'15115558 (107796'15115556) modify   bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 by client.7912580.0:24681012 2014-11-07 16:14:36.691882,107796'15115559 (107796'15115558) modify   bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 by client.7912580.0:24681219 2014-11-07 16:14:42.333104,107796'15115560 (107796'15115549) modify   a38c1b5a/rbd_data.7dfe93912c40a.00000000000000c2/head//6 by client.14531879.0:688901 2014-11-07 16:14:29.556680
  -276> 2014-11-07 16:14:56.883551 7f8606e45700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'15115562, trimmed: 107712'15112561
  -275> 2014-11-07 16:14:56.883602 7f8606e45700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.35a 0x5bbbbc0)/0x5bbbbc0
  -274> 2014-11-07 16:14:56.883613 7f8606e45700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724753 0x26334d80
  -273> 2014-11-07 16:14:56.883623 7f8606e45700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.883623, event: commit_queued_for_journal_write, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -272> 2014-11-07 16:14:56.883686 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] eval_repop repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=0 applied?=0 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4) wants=ad
  -271> 2014-11-07 16:14:56.883755 7f8606e45700 10 osd.21 107796 dequeue_op 0x20da0e00 finish
  -270> 2014-11-07 16:14:56.883776 7f8606e45700 10 osd.21 107796 dequeue_op 0x217db500 prio 127 cost 10773 latency 1.350910 osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]
  -269> 2014-11-07 16:14:56.883797 7f8606e45700  5 -- op tracker -- seq: 11664, time: 2014-11-07 16:14:56.883797, event: reached_pg, op: osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
  -268> 2014-11-07 16:14:56.883857 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] handle_message: 0x217db500
  -267> 2014-11-07 16:14:56.883874 7f8606e45700 15 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_sub_op osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v11
  -266> 2014-11-07 16:14:56.883897 7f8606e45700  7 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sub_op_scrub_map
  -265> 2014-11-07 16:14:56.883902 7f861cbdb700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.883902, event: write_thread_in_journal_buffer, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -264> 2014-11-07 16:14:56.883911 7f8606e45700  5 -- op tracker -- seq: 11664, time: 2014-11-07 16:14:56.883911, event: started, op: osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
  -263> 2014-11-07 16:14:56.883936 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  got 77 scrub map
  -262> 2014-11-07 16:14:56.884083 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] map version is 107796'15115561
  -261> 2014-11-07 16:14:56.884120 7f8606e45700 10 osd.21 107796 dequeue_op 0x217db500 finish
  -260> 2014-11-07 16:14:56.884136 7f8606e45700 10 osd.21 107796 dequeue_op 0x392afb00 prio 127 cost 4916 latency 0.301827 osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.576( v 107796'17462484 (107765'17459483,107796'17462484] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462479 lcod 107796'17462483 active]
  -259> 2014-11-07 16:14:56.884160 7f8606e45700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.884160, event: reached_pg, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -258> 2014-11-07 16:14:56.884147 7f8604640700 10 osd.21 107796 dequeue_op 0x289f3400 prio 127 cost 10773 latency 1.098847 osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]
  -257> 2014-11-07 16:14:56.884184 7f8604640700  5 -- op tracker -- seq: 11665, time: 2014-11-07 16:14:56.884184, event: reached_pg, op: osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
  -256> 2014-11-07 16:14:56.884173 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462484 (107765'17459483,107796'17462484] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462479 lcod 107796'17462483 active] handle_message: 0x392afb00
  -255> 2014-11-07 16:14:56.884206 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462484 (107765'17459483,107796'17462484] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462479 lcod 107796'17462483 active] sub_op_modify trans c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 v 107796'17462485 (transaction) 175
  -254> 2014-11-07 16:14:56.884206 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] handle_message: 0x289f3400
  -253> 2014-11-07 16:14:56.884224 7f8606e45700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.884224, event: started, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -252> 2014-11-07 16:14:56.884228 7f8604640700 15 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] do_sub_op osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[]) v11
  -251> 2014-11-07 16:14:56.884244 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462484 (107765'17459483,107796'17462484] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462479 lcod 107796'17462483 active] append_log log((107765'17459483,107796'17462484], crt=107796'17462479) [107796'17462485 (107796'17462435) modify   c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 by client.7868425.0:900145 2014-11-07 16:14:56.589649]
  -250> 2014-11-07 16:14:56.884252 7f8604640700  7 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sub_op_scrub_map
  -249> 2014-11-07 16:14:56.884277 7f8604640700  5 -- op tracker -- seq: 11665, time: 2014-11-07 16:14:56.884277, event: started, op: osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
  -248> 2014-11-07 16:14:56.884272 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459483,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462479 lcod 107796'17462483 active] add_log_entry 107796'17462485 (107796'17462435) modify   c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 by client.7868425.0:900145 2014-11-07 16:14:56.589649
  -247> 2014-11-07 16:14:56.884291 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  got 61 scrub map
  -246> 2014-11-07 16:14:56.884325 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459483,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] append_log  adding 1 keys
  -245> 2014-11-07 16:14:56.884341 7f8606e45700 10 trim log((107765'17459483,107796'17462485], crt=107796'17462483) to 107765'17459484
  -244> 2014-11-07 16:14:56.884348 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459484,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] append_log: trimming to 107796'17462483 entries 107796'17462480 (107796'17462479) modify   5cfafd76/rbd_data.47c9292ae8944a.000000000000091b/head//6 by client.7867357.0:22011389 2014-11-07 16:14:53.637296,107796'17462481 (107796'17462369) modify   e50cb576/rbd_data.d673c97e42c99d.0000000000002e2b/head//6 by client.14073900.0:16578237 2014-11-07 16:14:54.971398,107796'17462482 (107796'17462481) modify   e50cb576/rbd_data.d673c97e42c99d.0000000000002e2b/head//6 by client.14073900.0:16578238 2014-11-07 16:14:54.971482,107796'17462483 (107796'17462482) modify   e50cb576/rbd_data.d673c97e42c99d.0000000000002e2b/head//6 by client.14073900.0:16578239 2014-11-07 16:14:54.971646
  -243> 2014-11-07 16:14:56.884441 7f8606e45700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'17462485, trimmed: 107765'17459484
  -242> 2014-11-07 16:14:56.884457 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] map version is 107796'15115561
  -241> 2014-11-07 16:14:56.884484 7f8604640700 10 osd.21 107796 dequeue_op 0x289f3400 finish
  -240> 2014-11-07 16:14:56.884476 7f8606e45700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.884476, event: started, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -239> 2014-11-07 16:14:56.884495 7f8606e45700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.576 0x8621a70)/0x8621a70
  -238> 2014-11-07 16:14:56.884507 7f8606e45700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724754 0x20f26478
  -237> 2014-11-07 16:14:56.884503 7f8604640700  5 -- op tracker -- seq: 11665, time: 2014-11-07 16:14:56.884503, event: done, op: osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
  -236> 2014-11-07 16:14:56.884535 7f8604640700  5 -- op tracker -- seq: 11664, time: 2014-11-07 16:14:56.884535, event: done, op: osd_sub_op(unknown.0.0:0 6.35a 0//0//-1 [scrub-map] v 0'0 snapset=0=[]:[] snapc=0=[])
  -235> 2014-11-07 16:14:56.884514 7f8606e45700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.884514, event: commit_queued_for_journal_write, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -234> 2014-11-07 16:14:56.884579 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.35a at 0.000000, explicitly requested
  -233> 2014-11-07 16:14:56.884596 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.4d1 at 0.000000, explicitly requested
  -232> 2014-11-07 16:14:56.884601 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -231> 2014-11-07 16:14:56.884573 7f8606e45700 15 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459484,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] do_sub_op osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[]) v11
  -230> 2014-11-07 16:14:56.884612 7f8606e45700 10 osd.21 107796 dequeue_op 0x392afb00 finish
  -229> 2014-11-07 16:14:56.884605 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.4d1( v 107796'13096448 (107719'13093447,107796'13096448] local-les=107795 n=8950 ec=164 les/c 107795/107796 107793/107793/107793) [21,6,44] r=0 lpr=107793 crt=107796'13096446 lcod 107796'13096447 mlcod 107796'13096447 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -228> 2014-11-07 16:14:56.884628 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.4ee at 0.000000, explicitly requested
  -227> 2014-11-07 16:14:56.884634 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.550 at 0.000000, explicitly requested
  -226> 2014-11-07 16:14:56.884641 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.57c at 0.000000, explicitly requested
  -225> 2014-11-07 16:14:56.884644 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -224> 2014-11-07 16:14:56.884646 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.57c( v 107796'15283213 (107723'15280212,107796'15283213] local-les=107794 n=9148 ec=164 les/c 107794/107796 107793/107793/107793) [21,75,110] r=0 lpr=107793 crt=107796'15283211 lcod 107796'15283212 mlcod 107796'15283212 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -223> 2014-11-07 16:14:56.884666 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.58e at 0.000000, explicitly requested
  -222> 2014-11-07 16:14:56.884641 7f8604640700 10 osd.21 107796 dequeue_op 0x23fe9000 prio 127 cost 4916 latency 0.301689 osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.576( v 107796'17462485 (107765'17459484,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active]
  -221> 2014-11-07 16:14:56.884670 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -220> 2014-11-07 16:14:56.884672 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.58e( v 107796'11688757 (107576'11685756,107796'11688757] local-les=107794 n=9188 ec=164 les/c 107794/107796 107793/107793/107793) [21,43,7] r=0 lpr=107793 crt=107796'11688755 lcod 107796'11688756 mlcod 107796'11688756 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -219> 2014-11-07 16:14:56.884685 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.5a8 at 0.000000, explicitly requested
  -218> 2014-11-07 16:14:56.884689 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.5ca at 0.000000, explicitly requested
  -217> 2014-11-07 16:14:56.884672 7f8604640700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.884672, event: reached_pg, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -216> 2014-11-07 16:14:56.884693 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.62b at 0.000000, explicitly requested
  -215> 2014-11-07 16:14:56.884698 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.665 at 0.000000, explicitly requested
  -214> 2014-11-07 16:14:56.884700 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -213> 2014-11-07 16:14:56.884694 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459484,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] handle_message: 0x23fe9000
  -212> 2014-11-07 16:14:56.884702 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.665( v 107796'10195099 (107712'10191911,107796'10195099] local-les=107794 n=9212 ec=164 les/c 107794/107796 107793/107793/107793) [21,7,55] r=0 lpr=107793 crt=107792'10195060 mlcod 107796'10195098 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -211> 2014-11-07 16:14:56.884717 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.690 at 0.000000, explicitly requested
  -210> 2014-11-07 16:14:56.884720 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -209> 2014-11-07 16:14:56.884710 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459484,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] sub_op_modify trans c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 v 107796'17462486 (transaction) 175
  -208> 2014-11-07 16:14:56.884722 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.690( v 107796'29347459 (107769'29344458,107796'29347459] local-les=107794 n=9005 ec=164 les/c 107794/107796 107793/107793/107793) [21,102,123] r=0 lpr=107793 crt=107796'29347453 lcod 107796'29347458 mlcod 107796'29347458 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -207> 2014-11-07 16:14:56.884737 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.755 at 0.000000, explicitly requested
  -206> 2014-11-07 16:14:56.884743 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.773 at 0.000000, explicitly requested
  -205> 2014-11-07 16:14:56.884734 7f8604640700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.884734, event: started, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -204> 2014-11-07 16:14:56.884746 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.7e9 at 0.000000, explicitly requested
  -203> 2014-11-07 16:14:56.884757 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -202> 2014-11-07 16:14:56.884760 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.7e9( v 107796'8698746 (107712'8695745,107796'8698746] local-les=107794 n=9103 ec=164 les/c 107794/107796 107793/107793/107793) [21,28,4] r=0 lpr=107793 crt=107796'8698744 lcod 107796'8698745 mlcod 107796'8698745 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -201> 2014-11-07 16:14:56.884780 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.872 at 0.000000, explicitly requested
  -200> 2014-11-07 16:14:56.884785 7f861f894700 20 osd.21 107796 inc_scrubs_pending 0 + 1 active >= max 1
  -199> 2014-11-07 16:14:56.884788 7f861f894700 20 osd.21 pg_epoch: 107796 pg[6.872( v 107796'10222932 (107576'10219931,107796'10222932] local-les=107794 n=4540 ec=164 les/c 107794/107796 107793/107793/107793) [21,93,122] r=0 lpr=107793 crt=107796'10222930 lcod 107796'10222931 mlcod 107796'10222931 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sched_scrub: failed to reserve locally
  -198> 2014-11-07 16:14:56.884810 7f861f894700 10 osd.21 107796 sched_scrub scrubbing 6.99c at 0.000000, explicitly requested
  -197> 2014-11-07 16:14:56.884772 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462485 (107765'17459484,107796'17462485] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] append_log log((107765'17459484,107796'17462485], crt=107796'17462483) [107796'17462486 (107796'17462485) modify   c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 by client.7868425.0:900146 2014-11-07 16:14:56.589749]
  -196> 2014-11-07 16:14:56.884846 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459484,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] add_log_entry 107796'17462486 (107796'17462485) modify   c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 by client.7868425.0:900146 2014-11-07 16:14:56.589749
  -195> 2014-11-07 16:14:56.884873 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459484,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] append_log  adding 1 keys
  -194> 2014-11-07 16:14:56.884889 7f8604640700 10 trim log((107765'17459484,107796'17462486], crt=107796'17462483) to 107765'17459485
  -193> 2014-11-07 16:14:56.884895 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] append_log: trimming to 107796'17462483 entries 
  -192> 2014-11-07 16:14:56.884929 7f8604640700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'17462486, trimmed: 107765'17459485
  -191> 2014-11-07 16:14:56.884960 7f8604640700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.884960, event: started, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -190> 2014-11-07 16:14:56.884981 7f8604640700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.576 0x8621a70)/0x8621a70
  -189> 2014-11-07 16:14:56.884991 7f8604640700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724755 0x201970f8
  -188> 2014-11-07 16:14:56.885001 7f8604640700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.885001, event: commit_queued_for_journal_write, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -187> 2014-11-07 16:14:56.885022 7f8604640700 15 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] do_sub_op osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[]) v11
  -186> 2014-11-07 16:14:56.885050 7f8604640700 10 osd.21 107796 dequeue_op 0x23fe9000 finish
  -185> 2014-11-07 16:14:56.885801 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0xf4c83c0 seq 5724753 osr(6.35a 0x5bbbbc0) 0x26334d80
  -184> 2014-11-07 16:14:56.885790 7f861c3da700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.885790, event: journaled_completion_queued, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -183> 2014-11-07 16:14:56.885833 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0xf4c83c0 seq 5724753 osr(6.35a 0x5bbbbc0) 1939803 bytes   (queue has 3 ops and 1952771 bytes)
  -182> 2014-11-07 16:14:56.885849 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x26b98460
  -181> 2014-11-07 16:14:56.885864 7f8619bd5700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0xf4c83c0 seq 5724753 osr(6.35a 0x5bbbbc0)/0x5bbbbc0 start
  -180> 2014-11-07 16:14:56.885873 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x26334d80
  -179> 2014-11-07 16:14:56.885885 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6bdf7b/pglog_6.35a/0//-1
  -178> 2014-11-07 16:14:56.885880 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] op_commit: 1754
  -177> 2014-11-07 16:14:56.885888 7f861cbdb700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.885888, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -176> 2014-11-07 16:14:56.885916 7f86183d2700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.885916, event: op_commit, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
  -175> 2014-11-07 16:14:56.885949 7f8619bd5700 10 filestore oid: 6bdf7b/pglog_6.35a/0//-1 not skipping op, *spos 5724753.0.0
  -174> 2014-11-07 16:14:56.885953 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -173> 2014-11-07 16:14:56.885942 7f861cbdb700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.885942, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -172> 2014-11-07 16:14:56.885984 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -171> 2014-11-07 16:14:56.886010 7f8619bd5700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724753.0.1
  -170> 2014-11-07 16:14:56.886021 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -169> 2014-11-07 16:14:56.886065 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6bdf7b/pglog_6.35a/0//-1
  -168> 2014-11-07 16:14:56.886104 7f8619bd5700 10 filestore oid: 6bdf7b/pglog_6.35a/0//-1 not skipping op, *spos 5724753.0.2
  -167> 2014-11-07 16:14:56.886126 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -166> 2014-11-07 16:14:56.886154 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6bdf7b/pglog_6.35a/0//-1
  -165> 2014-11-07 16:14:56.886185 7f8619bd5700 10 filestore oid: 6bdf7b/pglog_6.35a/0//-1 not skipping op, *spos 5724753.0.3
  -164> 2014-11-07 16:14:56.886194 7f8619bd5700 10 filestore  > header.spos 0.0.0
  -163> 2014-11-07 16:14:56.886222 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 object_size 4194304 write_size 4194304
  -162> 2014-11-07 16:14:56.886229 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x9aa1870 seq 5724754 osr(6.576 0x8621a70) 0x20f26478
  -161> 2014-11-07 16:14:56.886236 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
  -160> 2014-11-07 16:14:56.886240 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x9aa1870 seq 5724754 osr(6.576 0x8621a70) 6484 bytes   (queue has 3 ops and 1952771 bytes)
  -159> 2014-11-07 16:14:56.886240 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 object_size 4194304 write_size 4194304 = -95
  -158> 2014-11-07 16:14:56.886251 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 2256896~1937408
  -157> 2014-11-07 16:14:56.886257 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x219fc1a0
  -156> 2014-11-07 16:14:56.886218 7f861c3da700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.886218, event: journaled_completion_queued, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -155> 2014-11-07 16:14:56.886271 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x9aa1f00 seq 5724755 osr(6.576 0x8621a70) 0x201970f8
  -154> 2014-11-07 16:14:56.886265 7f861c3da700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.886265, event: journaled_completion_queued, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -153> 2014-11-07 16:14:56.886277 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x9aa1f00 seq 5724755 osr(6.576 0x8621a70) 6484 bytes   (queue has 3 ops and 1952771 bytes)
  -152> 2014-11-07 16:14:56.886283 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x20755300
  -151> 2014-11-07 16:14:56.886299 7f86193d4700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x9aa1870 seq 5724754 osr(6.576 0x8621a70)/0x8621a70 start
  -150> 2014-11-07 16:14:56.886282 7f86183d2700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.886282, event: commit_sent, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -149> 2014-11-07 16:14:56.886307 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x20f26478
  -148> 2014-11-07 16:14:56.886316 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b672a/pglog_6.576/0//-1
  -147> 2014-11-07 16:14:56.886309 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462483 active] sub_op_modify_commit on op osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.85
  -146> 2014-11-07 16:14:56.886349 7f86183d2700 20 osd.21 107796 share_map_peer 0x1fdd39a0 already has epoch 107796
  -145> 2014-11-07 16:14:56.886359 7f86193d4700 10 filestore oid: 6b672a/pglog_6.576/0//-1 not skipping op, *spos 5724754.0.0
  -144> 2014-11-07 16:14:56.886363 7f86193d4700 10 filestore  > header.spos 0.0.0
  -143> 2014-11-07 16:14:56.886354 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.7:6801/27520 -- osd_sub_op_reply(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] ondisk, result = 0) v2 -- ?+0 0x219ca600 con 0x1fdd39a0
  -142> 2014-11-07 16:14:56.886410 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -141> 2014-11-07 16:14:56.886394 7f86183d2700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.886394, event: commit_sent, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
  -140> 2014-11-07 16:14:56.886435 7f86193d4700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724754.0.1
  -139> 2014-11-07 16:14:56.886439 7f86193d4700 10 filestore  > header.spos 0.0.0
  -138> 2014-11-07 16:14:56.886418 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462484 active] sub_op_modify_commit on op osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.85
  -137> 2014-11-07 16:14:56.886501 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6b672a/pglog_6.576/0//-1
  -136> 2014-11-07 16:14:56.886509 7f86183d2700 20 osd.21 107796 share_map_peer 0x1fdd39a0 already has epoch 107796
  -135> 2014-11-07 16:14:56.886528 7f86193d4700 10 filestore oid: 6b672a/pglog_6.576/0//-1 not skipping op, *spos 5724754.0.2
  -134> 2014-11-07 16:14:56.886531 7f86193d4700 10 filestore  > header.spos 0.0.0
  -133> 2014-11-07 16:14:56.886518 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.7:6801/27520 -- osd_sub_op_reply(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] ondisk, result = 0) v2 -- ?+0 0x2114a080 con 0x1fdd39a0
  -132> 2014-11-07 16:14:56.886550 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b672a/pglog_6.576/0//-1
  -131> 2014-11-07 16:14:56.886572 7f86193d4700 10 filestore oid: 6b672a/pglog_6.576/0//-1 not skipping op, *spos 5724754.0.3
  -130> 2014-11-07 16:14:56.886575 7f86193d4700 10 filestore  > header.spos 0.0.0
  -129> 2014-11-07 16:14:56.886601 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 object_size 4194304 write_size 4194304
  -128> 2014-11-07 16:14:56.886662 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
  -127> 2014-11-07 16:14:56.886675 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 object_size 4194304 write_size 4194304 = -95
  -126> 2014-11-07 16:14:56.886684 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 36352~4096
  -125> 2014-11-07 16:14:56.886723 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 36352~4096 = 4096
  -124> 2014-11-07 16:14:56.886757 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting '_'
  -123> 2014-11-07 16:14:56.886772 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting 'snapset'
  -122> 2014-11-07 16:14:56.886777 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6
  -121> 2014-11-07 16:14:56.886811 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 = 0
  -120> 2014-11-07 16:14:56.886839 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting '_'
  -119> 2014-11-07 16:14:56.886851 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting 'snapset'
  -118> 2014-11-07 16:14:56.886856 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6
  -117> 2014-11-07 16:14:56.886880 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 = 0
  -116> 2014-11-07 16:14:56.886891 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x9aa1870 seq 5724754 r = 0, finisher 0x264b42a0 0
  -115> 2014-11-07 16:14:56.886895 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x9aa1870 seq 5724754 osr(6.576 0x8621a70)/0x8621a70
  -114> 2014-11-07 16:14:56.886913 7f86193d4700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x9aa1f00 seq 5724755 osr(6.576 0x8621a70)/0x8621a70 start
  -113> 2014-11-07 16:14:56.886917 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x201970f8
  -112> 2014-11-07 16:14:56.886922 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b672a/pglog_6.576/0//-1
  -111> 2014-11-07 16:14:56.886930 7f8618bd3700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.886930, event: sub_op_applied, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -110> 2014-11-07 16:14:56.886951 7f86193d4700 10 filestore oid: 6b672a/pglog_6.576/0//-1 not skipping op, *spos 5724755.0.0
  -109> 2014-11-07 16:14:56.886954 7f86193d4700 10 filestore  > header.spos 0.0.0
  -108> 2014-11-07 16:14:56.886954 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462485 active] sub_op_modify_applied on 0x20f26380 op osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[]) v11
  -107> 2014-11-07 16:14:56.886988 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
  -106> 2014-11-07 16:14:56.886980 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462485 active] op_applied version 107796'17462485
  -105> 2014-11-07 16:14:56.887008 7f8618bd3700  5 -- op tracker -- seq: 11687, time: 2014-11-07 16:14:56.887008, event: done, op: osd_sub_op(client.7868425.0:900145 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462485 snapset=0=[]:[] snapc=0=[])
  -104> 2014-11-07 16:14:56.887029 7f86193d4700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724755.0.1
  -103> 2014-11-07 16:14:56.887040 7f86193d4700 10 filestore  > header.spos 0.0.0
  -102> 2014-11-07 16:14:56.887083 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6b672a/pglog_6.576/0//-1
  -101> 2014-11-07 16:14:56.887127 7f86193d4700 10 filestore oid: 6b672a/pglog_6.576/0//-1 not skipping op, *spos 5724755.0.2
  -100> 2014-11-07 16:14:56.887136 7f86193d4700 10 filestore  > header.spos 0.0.0
   -99> 2014-11-07 16:14:56.887156 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b672a/pglog_6.576/0//-1
   -98> 2014-11-07 16:14:56.887179 7f86193d4700 10 filestore oid: 6b672a/pglog_6.576/0//-1 not skipping op, *spos 5724755.0.3
   -97> 2014-11-07 16:14:56.887186 7f86193d4700 10 filestore  > header.spos 0.0.0
   -96> 2014-11-07 16:14:56.887216 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 object_size 4194304 write_size 4194304
   -95> 2014-11-07 16:14:56.887228 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
   -94> 2014-11-07 16:14:56.887224 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 2256896~1937408 = 1937408
   -93> 2014-11-07 16:14:56.887231 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 object_size 4194304 write_size 4194304 = -95
   -92> 2014-11-07 16:14:56.887237 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 60928~4096
   -91> 2014-11-07 16:14:56.887260 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 257 getting '_'
   -90> 2014-11-07 16:14:56.887264 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 60928~4096 = 4096
   -89> 2014-11-07 16:14:56.887270 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 257 getting 'snapset'
   -88> 2014-11-07 16:14:56.887276 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6
   -87> 2014-11-07 16:14:56.887292 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting '_'
   -86> 2014-11-07 16:14:56.887308 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 = 0
   -85> 2014-11-07 16:14:56.887310 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting 'snapset'
   -84> 2014-11-07 16:14:56.887320 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6
   -83> 2014-11-07 16:14:56.887330 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 257 getting '_'
   -82> 2014-11-07 16:14:56.887338 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 257 getting 'snapset'
   -81> 2014-11-07 16:14:56.887343 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6
   -80> 2014-11-07 16:14:56.887350 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 = 0
   -79> 2014-11-07 16:14:56.887366 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.35a_head/bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 = 0
   -78> 2014-11-07 16:14:56.887373 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0xf4c83c0 seq 5724753 r = 0, finisher 0x22265640 0xff56380
   -77> 2014-11-07 16:14:56.887377 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0xf4c83c0 seq 5724753 osr(6.35a 0x5bbbbc0)/0x5bbbbc0
   -76> 2014-11-07 16:14:56.887379 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting '_'
   -75> 2014-11-07 16:14:56.887391 7f86193d4700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 632 getting 'snapset'
   -74> 2014-11-07 16:14:56.887395 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6
   -73> 2014-11-07 16:14:56.887421 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.576_head/c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 = 0
   -72> 2014-11-07 16:14:56.887428 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x9aa1f00 seq 5724755 r = 0, finisher 0x263621e0 0
   -71> 2014-11-07 16:14:56.887409 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] op_applied: 1754
   -70> 2014-11-07 16:14:56.887432 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x9aa1f00 seq 5724755 osr(6.576 0x8621a70)/0x8621a70
   -69> 2014-11-07 16:14:56.887435 7f8618bd3700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.887435, event: op_applied, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
   -68> 2014-11-07 16:14:56.887451 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] op_applied version 107796'15115562
   -67> 2014-11-07 16:14:56.887483 7f8618bd3700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.887483, event: sub_op_applied, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
   -66> 2014-11-07 16:14:56.887504 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462485 active] sub_op_modify_applied on 0x20197000 op osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[]) v11
   -65> 2014-11-07 16:14:56.887526 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.576( v 107796'17462486 (107765'17459485,107796'17462486] local-les=107794 n=9026 ec=164 les/c 107794/107796 107793/107793/105195) [85,21,103] r=1 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107796'17462483 lcod 107796'17462485 active] op_applied version 107796'17462486
   -64> 2014-11-07 16:14:56.887549 7f8618bd3700  5 -- op tracker -- seq: 11688, time: 2014-11-07 16:14:56.887549, event: done, op: osd_sub_op(client.7868425.0:900146 6.576 c6b02d76/rbd_data.7c5a645302f7e.0000000000000000/head//6 [] v 107796'17462486 snapset=0=[]:[] snapc=0=[])
   -63> 2014-11-07 16:14:56.890393 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.9d8_head/805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6 '_' = 244
   -62> 2014-11-07 16:14:56.890427 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.9d8_head/805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6 'snapset'
   -61> 2014-11-07 16:14:56.892553 7f85f92da700  1 -- 172.30.5.2:6803/25692 <== osd.77 172.30.5.6:6841/32590 95 ==== osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2 ==== 178+0+0 (652048473 0 0) 0x28a5e080 con 0x1fe01580
   -60> 2014-11-07 16:14:56.892584 7f85f92da700  5 -- op tracker -- seq: 11700, time: 2014-11-07 16:14:56.892471, event: header_read, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -59> 2014-11-07 16:14:56.892597 7f85f92da700  5 -- op tracker -- seq: 11700, time: 2014-11-07 16:14:56.892473, event: throttled, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -58> 2014-11-07 16:14:56.892604 7f85f92da700  5 -- op tracker -- seq: 11700, time: 2014-11-07 16:14:56.892529, event: all_read, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -57> 2014-11-07 16:14:56.892611 7f85f92da700  5 -- op tracker -- seq: 11700, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -56> 2014-11-07 16:14:56.892618 7f85f92da700 10 osd.21 107796 handle_replica_op osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2 epoch 107796
   -55> 2014-11-07 16:14:56.892627 7f85f92da700 20 osd.21 107796 should_share_map osd.77 172.30.5.6:6841/32590 107796
   -54> 2014-11-07 16:14:56.892638 7f85f92da700 15 osd.21 107796 enqueue_op 0x217dbd00 prio 196 cost 0 latency 0.000167 osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2
   -53> 2014-11-07 16:14:56.892686 7f8606e45700 10 osd.21 107796 dequeue_op 0x217dbd00 prio 196 cost 0 latency 0.000215 osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2 pg pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]
   -52> 2014-11-07 16:14:56.892723 7f8606e45700  5 -- op tracker -- seq: 11700, time: 2014-11-07 16:14:56.892722, event: reached_pg, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -51> 2014-11-07 16:14:56.892736 7f8606e45700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] handle_message: 0x217dbd00
   -50> 2014-11-07 16:14:56.892759 7f8606e45700  5 -- op tracker -- seq: 11700, time: 2014-11-07 16:14:56.892759, event: started, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -49> 2014-11-07 16:14:56.892768 7f8606e45700  7 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sub_op_modify_reply: tid 1754 op  ack_type 4 from 77
   -48> 2014-11-07 16:14:56.892789 7f8606e45700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.892789, event: sub_op_commit_rec, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
   -47> 2014-11-07 16:14:56.892803 7f8606e45700 10 osd.21 107796 dequeue_op 0x217dbd00 finish
   -46> 2014-11-07 16:14:56.892807 7f8606e45700  5 -- op tracker -- seq: 11700, time: 2014-11-07 16:14:56.892807, event: done, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -45> 2014-11-07 16:14:56.900964 7f85fa9f1700  1 -- 172.30.5.2:6803/25692 <== osd.61 172.30.5.5:6831/12481 81 ==== osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2 ==== 178+0+0 (1830880513 0 0) 0x20fff580 con 0x1fe34b00
   -44> 2014-11-07 16:14:56.900995 7f85fa9f1700  5 -- op tracker -- seq: 11701, time: 2014-11-07 16:14:56.900890, event: header_read, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -43> 2014-11-07 16:14:56.901008 7f85fa9f1700  5 -- op tracker -- seq: 11701, time: 2014-11-07 16:14:56.900892, event: throttled, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -42> 2014-11-07 16:14:56.901015 7f85fa9f1700  5 -- op tracker -- seq: 11701, time: 2014-11-07 16:14:56.900949, event: all_read, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -41> 2014-11-07 16:14:56.901023 7f85fa9f1700  5 -- op tracker -- seq: 11701, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -40> 2014-11-07 16:14:56.901030 7f85fa9f1700 10 osd.21 107796 handle_replica_op osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2 epoch 107796
   -39> 2014-11-07 16:14:56.901046 7f85fa9f1700 20 osd.21 107796 should_share_map osd.61 172.30.5.5:6831/12481 107796
   -38> 2014-11-07 16:14:56.901056 7f85fa9f1700 15 osd.21 107796 enqueue_op 0x218ba400 prio 196 cost 0 latency 0.000165 osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2
   -37> 2014-11-07 16:14:56.901101 7f8604640700 10 osd.21 107796 dequeue_op 0x218ba400 prio 196 cost 0 latency 0.000211 osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0) v2 pg pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]
   -36> 2014-11-07 16:14:56.901145 7f8604640700  5 -- op tracker -- seq: 11701, time: 2014-11-07 16:14:56.901145, event: reached_pg, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -35> 2014-11-07 16:14:56.901176 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] handle_message: 0x218ba400
   -34> 2014-11-07 16:14:56.901200 7f8604640700  5 -- op tracker -- seq: 11701, time: 2014-11-07 16:14:56.901200, event: started, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -33> 2014-11-07 16:14:56.901215 7f8604640700  7 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] sub_op_modify_reply: tid 1754 op  ack_type 4 from 61
   -32> 2014-11-07 16:14:56.901232 7f8604640700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.901232, event: sub_op_commit_rec, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
   -31> 2014-11-07 16:14:56.901247 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] repop_all_applied: repop tid 1754 all applied 
   -30> 2014-11-07 16:14:56.901271 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] eval_repop repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=0 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4) wants=ad
   -29> 2014-11-07 16:14:56.901294 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  sending ack on repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=0 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4) 0x2631d2c0
   -28> 2014-11-07 16:14:56.901313 7f8604640700  1 -- 172.30.5.2:6802/25692 --> 172.30.1.96:0/1015303 -- osd_op_reply(24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] v107796'15115562 uv15115562 ack = 0) v6 -- ?+0 0x2631d2c0 con 0x38ed9a0
   -27> 2014-11-07 16:14:56.901340 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 luod=107796'15115561 crt=107796'15115560 lcod 107796'15115560 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] repop_all_committed: repop tid 1754 all committed 
   -26> 2014-11-07 16:14:56.901354 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] eval_repop repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=1 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4) wants=ad
   -25> 2014-11-07 16:14:56.901374 7f8604640700 15 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] log_op_stats osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4 inb 1937691 outb 0 rlat 3.288641 lat 3.288674
   -24> 2014-11-07 16:14:56.901424 7f8604640700 15 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] publish_stats_to_osd 107796:22234100
   -23> 2014-11-07 16:14:56.901441 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115560 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  sending commit on repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=1 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4) 0x2142c600
   -22> 2014-11-07 16:14:56.901459 7f8604640700  1 -- 172.30.5.2:6802/25692 --> 172.30.1.96:0/1015303 -- osd_op_reply(24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] v107796'15115562 uv15115562 ondisk = 0) v6 -- ?+0 0x2142c600 con 0x38ed9a0
   -21> 2014-11-07 16:14:56.901476 7f8604640700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.901476, event: commit_sent, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
   -20> 2014-11-07 16:14:56.901491 7f8604640700 10 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115561 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  removing repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=1 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4)
   -19> 2014-11-07 16:14:56.901510 7f8604640700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115561 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]    q front is repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=1 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4)
   -18> 2014-11-07 16:14:56.901527 7f8604640700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115561 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] remove_repop repgather(0x2226cb40 107796'15115562 rep_tid=1754 committed?=1 applied?=1 lock=0 op=osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796) v4)
   -17> 2014-11-07 16:14:56.901543 7f8604640700 20 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115561 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  obc obc(bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 rwstate(write n=1 w=0))
   -16> 2014-11-07 16:14:56.901556 7f8604640700 15 osd.21 pg_epoch: 107796 pg[6.35a( v 107796'15115562 (107712'15112561,107796'15115562] local-les=107794 n=9023 ec=164 les/c 107794/107796 107793/107793/107793) [21,61,77] r=0 lpr=107793 crt=107796'15115560 lcod 107796'15115561 mlcod 107796'15115561 active+clean+scrubbing+deep+repair MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB]  requeue_ops 
   -15> 2014-11-07 16:14:56.901577 7f8604640700  5 -- op tracker -- seq: 11562, time: 2014-11-07 16:14:56.901577, event: done, op: osd_op(client.7912580.0:24681633 rbd_data.74ae9c3be03aff.0000000000001188 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2256896~1937408] 6.bcca835a ack+ondisk+write e107796)
   -14> 2014-11-07 16:14:56.901607 7f8604640700 10 osd.21 107796 dequeue_op 0x218ba400 finish
   -13> 2014-11-07 16:14:56.901617 7f8604640700  5 -- op tracker -- seq: 11701, time: 2014-11-07 16:14:56.901617, event: done, op: osd_sub_op_reply(client.7912580.0:24681633 6.35a bcca835a/rbd_data.74ae9c3be03aff.0000000000001188/head//6 [] ondisk, result = 0)
   -12> 2014-11-07 16:14:56.902035 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.9d8_head/805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6 'snapset' = 31
   -11> 2014-11-07 16:14:56.902056 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] populate_obc_watchers 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6
   -10> 2014-11-07 16:14:56.902083 7f8602e3d700 20 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] ReplicatedPG::check_blacklisted_obc_watchers for obc 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6
    -9> 2014-11-07 16:14:56.902099 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] get_object_context: creating obc from disk: 0x2170db80
    -8> 2014-11-07 16:14:56.902143 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] get_object_context: 0x2170db80 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6 rwstate(none n=0 w=0) oi: 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6(81189'5583284 client.5259509.0:8841401 [1907] dirty s 3776512 uv5543997) ssc: 0x217021c0 snapset: 0=[]:[]+head
    -7> 2014-11-07 16:14:56.902181 7f8602e3d700 15 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.9d8_head/805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6 '_'
    -6> 2014-11-07 16:14:56.902217 7f8602e3d700 10 filestore(/var/lib/ceph/osd/cloud-21) getattr 6.9d8_head/805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6 '_' = 267
    -5> 2014-11-07 16:14:56.902236 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] populate_obc_watchers 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6
    -4> 2014-11-07 16:14:56.902283 7f8602e3d700 20 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] ReplicatedPG::check_blacklisted_obc_watchers for obc 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6
    -3> 2014-11-07 16:14:56.902311 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] get_object_context: creating obc from disk: 0x2115c840
    -2> 2014-11-07 16:14:56.902326 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] get_object_context: 0x2115c840 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6 rwstate(none n=0 w=0) oi: 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/head//6(106376'7966817 client.12931724.0:1 wrlock_by=unknown.0.0:0 dirty s 4194304 uv7966817) ssc: 0x217021c0 snapset: 0=[]:[]+head
    -1> 2014-11-07 16:14:56.902354 7f8602e3d700 10 osd.21 pg_epoch: 107796 pg[6.9d8( v 107796'8021899 (107558'8016433,107796'8021899] local-les=107795 n=4517 ec=164 les/c 107795/107796 107793/107793/107793) [21,93,49] r=0 lpr=107793 crt=107792'8021894 mlcod 107796'8021898 active+clean MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB snaptrimq=[1907~1,1937~3,193e~2,1941~6,1978~1,19ef~2,19f2~1,19f4~3,19fa~5,1a01~5,1a09~17]] 805979d8/rbd_data.43c0b25a5d8819.0000000000000185/1907//6 old_snaps 1907 old snapset 0=[]:[]+head
     0> 2014-11-07 16:14:56.905122 7f8602e3d700 -1 osd/ReplicatedPG.cc: In function 'ReplicatedPG::RepGather* ReplicatedPG::trim_object(const hobject_t&)' thread 7f8602e3d700 time 2014-11-07 16:14:56.902373
osd/ReplicatedPG.cc: 2430: FAILED assert(snapset.seq)

 ceph version 0.87-6-gdba7def (dba7defc623474ad17263c9fccfec60fe7a439f0)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xa9feaf]
 2: (ReplicatedPG::trim_object(hobject_t const&)+0x222d) [0x8139ed]
 3: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim const&)+0x43e) [0x82b9be]
 4: (boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xc0) [0x870ce0]
 5: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0xfb) [0x85618b]
 6: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x1e) [0x85633e]
 7: (ReplicatedPG::snap_trimmer()+0x4f8) [0x7d5ef8]
 8: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x673ab4]
 9: (ThreadPool::worker(ThreadPool::WorkThread*)+0x48e) [0xa8fade]
 10: (ThreadPool::WorkThread::entry()+0x10) [0xa92870]
 11: (()+0x7e9a) [0x7f8626431e9a]
 12: (clone()+0x6d) [0x7f8624dfc31d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
  20/20 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   1/ 1 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.21.log
--- end dump of recent events ---
2014-11-07 16:14:56.935001 7f860d652700  1 -- 172.30.5.2:6804/25692 <== osd.31 172.30.5.3:0/20618 45 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:56.935867) v2 ==== 47+0+0 (760863541 0 0) 0x28a6a000 con 0x20c85420
2014-11-07 16:14:56.935053 7f860d652700  1 -- 172.30.5.2:6804/25692 --> 172.30.5.3:0/20618 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:56.935867) v2 -- ?+0 0x20464000 con 0x20c85420
2014-11-07 16:14:56.935117 7f860d652700 20 osd.21 107796 share_map_peer 0x1fe63580 already has epoch 107796
2014-11-07 16:14:56.935321 7f860ee55700  1 -- 172.30.5.2:6805/25692 <== osd.31 172.30.5.3:0/20618 45 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:56.935867) v2 ==== 47+0+0 (760863541 0 0) 0x213b6400 con 0x20af1160
2014-11-07 16:14:56.935362 7f860ee55700  1 -- 172.30.5.2:6805/25692 --> 172.30.5.3:0/20618 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:56.935867) v2 -- ?+0 0x22250800 con 0x20af1160
2014-11-07 16:14:56.935446 7f860ee55700 20 osd.21 107796 share_map_peer 0x1fe63580 already has epoch 107796
2014-11-07 16:14:57.005860 7f85f61a9700  1 -- 172.30.5.2:6803/25692 <== osd.13 172.30.5.1:6839/22281 212 ==== osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+344889 (2809630467 0 2077839927) 0x2199f000 con 0x1fe6a840
2014-11-07 16:14:57.005975 7f85f61a9700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
2014-11-07 16:14:57.005988 7f85f61a9700 20 osd.21 107796 should_share_map osd.13 172.30.5.1:6839/22281 107796
2014-11-07 16:14:57.006003 7f85f61a9700 15 osd.21 107796 enqueue_op 0x28b3a800 prio 127 cost 344889 latency 0.002168 osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11
2014-11-07 16:14:57.006064 7f8608648700 10 osd.21 107796 dequeue_op 0x28b3a800 prio 127 cost 344889 latency 0.002229 osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16]
2014-11-07 16:14:57.006171 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] handle_message: 0x28b3a800
2014-11-07 16:14:57.006237 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify trans b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 v 107796'32423868 (transaction) 176
2014-11-07 16:14:57.006278 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log log((107765'32418540,107796'32423867], crt=107792'32423690) [107796'32423868 (107796'32423845) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681747 2014-11-07 16:14:56.986326]
2014-11-07 16:14:57.006309 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] add_log_entry 107796'32423868 (107796'32423845) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681747 2014-11-07 16:14:56.986326
2014-11-07 16:14:57.006344 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log  adding 1 keys
2014-11-07 16:14:57.006361 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log: trimming to 107786'32421641 entries 
2014-11-07 16:14:57.006406 7f8608648700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'32423868, trimmed: 
2014-11-07 16:14:57.006451 7f8608648700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.456 0x7576250)/0x7576250
2014-11-07 16:14:57.006459 7f8608648700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724756 0x1eb3d0f8
2014-11-07 16:14:57.006500 7f8608648700 15 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] do_sub_op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11
2014-11-07 16:14:57.006519 7f8608648700 10 osd.21 107796 dequeue_op 0x28b3a800 finish
2014-11-07 16:14:57.007400 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x1a05e140 seq 5724756 osr(6.456 0x7576250) 0x1eb3d0f8
2014-11-07 16:14:57.007412 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x1a05e140 seq 5724756 osr(6.456 0x7576250) 346424 bytes   (queue has 1 ops and 346424 bytes)
2014-11-07 16:14:57.007429 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x214251c0
2014-11-07 16:14:57.007450 7f8619bd5700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x1a05e140 seq 5724756 osr(6.456 0x7576250)/0x7576250 start
2014-11-07 16:14:57.007461 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x1eb3d0f8
2014-11-07 16:14:57.007470 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b81fa/pglog_6.456/0//-1
2014-11-07 16:14:57.007497 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify_commit on op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.13
2014-11-07 16:14:57.007538 7f86183d2700 20 osd.21 107796 share_map_peer 0x1fe6a840 already has epoch 107796
2014-11-07 16:14:57.007542 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.1:6839/22281 -- osd_sub_op_reply(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] ondisk, result = 0) v2 -- ?+0 0x20f3c840 con 0x1fe6a840
2014-11-07 16:14:57.007563 7f8619bd5700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724756.0.0
2014-11-07 16:14:57.007567 7f8619bd5700 10 filestore  > header.spos 0.0.0
2014-11-07 16:14:57.007651 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
2014-11-07 16:14:57.007684 7f8619bd5700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724756.0.1
2014-11-07 16:14:57.007687 7f8619bd5700 10 filestore  > header.spos 0.0.0
2014-11-07 16:14:57.007724 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6b81fa/pglog_6.456/0//-1
2014-11-07 16:14:57.007785 7f8619bd5700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724756.0.2
2014-11-07 16:14:57.007791 7f8619bd5700 10 filestore  > header.spos 0.0.0
2014-11-07 16:14:57.007812 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b81fa/pglog_6.456/0//-1
2014-11-07 16:14:57.007842 7f8619bd5700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724756.0.3
2014-11-07 16:14:57.007845 7f8619bd5700 10 filestore  > header.spos 0.0.0
2014-11-07 16:14:57.007928 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 object_size 4194304 write_size 4194304
2014-11-07 16:14:57.008028 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
2014-11-07 16:14:57.008033 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 object_size 4194304 write_size 4194304 = -95
2014-11-07 16:14:57.008042 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 0~344064
2014-11-07 16:14:57.008312 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 0~344064 = 344064
2014-11-07 16:14:57.008349 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting '_'
2014-11-07 16:14:57.008391 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting 'snapset'
2014-11-07 16:14:57.008398 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6
2014-11-07 16:14:57.008431 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 = 0
2014-11-07 16:14:57.008454 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting '_'
2014-11-07 16:14:57.008462 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting 'snapset'
2014-11-07 16:14:57.008466 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6
2014-11-07 16:14:57.008530 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 = 0
2014-11-07 16:14:57.008543 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x1a05e140 seq 5724756 r = 0, finisher 0x255c0b80 0
2014-11-07 16:14:57.008554 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x1a05e140 seq 5724756 osr(6.456 0x7576250)/0x7576250
2014-11-07 16:14:57.008614 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify_applied on 0x1eb3d000 op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11
2014-11-07 16:14:57.008636 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] op_applied version 107796'32423868
2014-11-07 16:14:57.021968 7f860d652700  1 -- 172.30.5.2:6804/25692 <== osd.77 172.30.5.6:0/32590 48 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:57.020796) v2 ==== 47+0+0 (1385461384 0 0) 0x26468400 con 0x2074a2c0
2014-11-07 16:14:57.021988 7f860ee55700  1 -- 172.30.5.2:6805/25692 <== osd.77 172.30.5.6:0/32590 48 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:57.020796) v2 ==== 47+0+0 (1385461384 0 0) 0x21d18a00 con 0x204bf000
2014-11-07 16:14:57.022025 7f860d652700  1 -- 172.30.5.2:6804/25692 --> 172.30.5.6:0/32590 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:57.020796) v2 -- ?+0 0x28a6a000 con 0x2074a2c0
2014-11-07 16:14:57.022063 7f860d652700 20 osd.21 107796 share_map_peer 0x1fe01580 already has epoch 107796
2014-11-07 16:14:57.022088 7f860ee55700  1 -- 172.30.5.2:6805/25692 --> 172.30.5.6:0/32590 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:57.020796) v2 -- ?+0 0x213b6400 con 0x204bf000
2014-11-07 16:14:57.022164 7f860ee55700 20 osd.21 107796 share_map_peer 0x1fe01580 already has epoch 107796
2014-11-07 16:14:57.022237 7f85f61a9700  1 -- 172.30.5.2:6803/25692 <== osd.13 172.30.5.1:6839/22281 213 ==== osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+1049401 (252162102 0 3108292305) 0x20eb8800 con 0x1fe6a840
2014-11-07 16:14:57.022355 7f85f61a9700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
2014-11-07 16:14:57.022368 7f85f61a9700 20 osd.21 107796 should_share_map osd.13 172.30.5.1:6839/22281 107796
2014-11-07 16:14:57.022379 7f85f61a9700 15 osd.21 107796 enqueue_op 0x21fa0e00 prio 127 cost 1049401 latency 0.005196 osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11
2014-11-07 16:14:57.022427 7f8605e43700 10 osd.21 107796 dequeue_op 0x21fa0e00 prio 127 cost 1049401 latency 0.005243 osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16]
2014-11-07 16:14:57.022488 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] handle_message: 0x21fa0e00
2014-11-07 16:14:57.022509 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify trans b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 v 107796'32423869 (transaction) 176
2014-11-07 16:14:57.022546 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log log((107765'32418540,107796'32423868], crt=107792'32423690) [107796'32423869 (107796'32423868) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681748 2014-11-07 16:14:56.986423]
2014-11-07 16:14:57.022584 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] add_log_entry 107796'32423869 (107796'32423868) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681748 2014-11-07 16:14:56.986423
2014-11-07 16:14:57.022617 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log  adding 1 keys
2014-11-07 16:14:57.022633 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log: trimming to 107786'32421641 entries 
2014-11-07 16:14:57.022673 7f8605e43700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'32423869, trimmed: 
2014-11-07 16:14:57.022738 7f8605e43700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.456 0x7576250)/0x7576250
2014-11-07 16:14:57.022749 7f8605e43700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724757 0x1595fb78
2014-11-07 16:14:57.022796 7f8605e43700 15 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] do_sub_op osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11
2014-11-07 16:14:57.022847 7f8605e43700 10 osd.21 107796 dequeue_op 0x21fa0e00 finish
2014-11-07 16:14:57.024605 7f8602e3d700 -1 *** Caught signal (Aborted) **
 in thread 7f8602e3d700

 ceph version 0.87-6-gdba7def (dba7defc623474ad17263c9fccfec60fe7a439f0)
 1: /usr/bin/ceph-osd() [0x9b6725]
 2: (()+0xfcb0) [0x7f8626439cb0]
 3: (gsignal()+0x35) [0x7f8624d3e0d5]
 4: (abort()+0x17b) [0x7f8624d4183b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f862569069d]
 6: (()+0xb5846) [0x7f862568e846]
 7: (()+0xb5873) [0x7f862568e873]
 8: (()+0xb596e) [0x7f862568e96e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xaa0089]
 10: (ReplicatedPG::trim_object(hobject_t const&)+0x222d) [0x8139ed]
 11: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim const&)+0x43e) [0x82b9be]
 12: (boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xc0) [0x870ce0]
 13: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0xfb) [0x85618b]
 14: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x1e) [0x85633e]
 15: (ReplicatedPG::snap_trimmer()+0x4f8) [0x7d5ef8]
 16: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x673ab4]
 17: (ThreadPool::worker(ThreadPool::WorkThread*)+0x48e) [0xa8fade]
 18: (ThreadPool::WorkThread::entry()+0x10) [0xa92870]
 19: (()+0x7e9a) [0x7f8626431e9a]
 20: (clone()+0x6d) [0x7f8624dfc31d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -103> 2014-11-07 16:14:56.935001 7f860d652700  1 -- 172.30.5.2:6804/25692 <== osd.31 172.30.5.3:0/20618 45 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:56.935867) v2 ==== 47+0+0 (760863541 0 0) 0x28a6a000 con 0x20c85420
  -102> 2014-11-07 16:14:56.935053 7f860d652700  1 -- 172.30.5.2:6804/25692 --> 172.30.5.3:0/20618 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:56.935867) v2 -- ?+0 0x20464000 con 0x20c85420
  -101> 2014-11-07 16:14:56.935117 7f860d652700 20 osd.21 107796 share_map_peer 0x1fe63580 already has epoch 107796
  -100> 2014-11-07 16:14:56.935321 7f860ee55700  1 -- 172.30.5.2:6805/25692 <== osd.31 172.30.5.3:0/20618 45 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:56.935867) v2 ==== 47+0+0 (760863541 0 0) 0x213b6400 con 0x20af1160
   -99> 2014-11-07 16:14:56.935362 7f860ee55700  1 -- 172.30.5.2:6805/25692 --> 172.30.5.3:0/20618 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:56.935867) v2 -- ?+0 0x22250800 con 0x20af1160
   -98> 2014-11-07 16:14:56.935446 7f860ee55700 20 osd.21 107796 share_map_peer 0x1fe63580 already has epoch 107796
   -97> 2014-11-07 16:14:57.005860 7f85f61a9700  1 -- 172.30.5.2:6803/25692 <== osd.13 172.30.5.1:6839/22281 212 ==== osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+344889 (2809630467 0 2077839927) 0x2199f000 con 0x1fe6a840
   -96> 2014-11-07 16:14:57.005894 7f85f61a9700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.003834, event: header_read, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -95> 2014-11-07 16:14:57.005928 7f85f61a9700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.003836, event: throttled, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -94> 2014-11-07 16:14:57.005937 7f85f61a9700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.005836, event: all_read, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -93> 2014-11-07 16:14:57.005962 7f85f61a9700  5 -- op tracker -- seq: 11702, time: 0.000000, event: dispatched, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -92> 2014-11-07 16:14:57.005975 7f85f61a9700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
   -91> 2014-11-07 16:14:57.005988 7f85f61a9700 20 osd.21 107796 should_share_map osd.13 172.30.5.1:6839/22281 107796
   -90> 2014-11-07 16:14:57.006003 7f85f61a9700 15 osd.21 107796 enqueue_op 0x28b3a800 prio 127 cost 344889 latency 0.002168 osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11
   -89> 2014-11-07 16:14:57.006064 7f8608648700 10 osd.21 107796 dequeue_op 0x28b3a800 prio 127 cost 344889 latency 0.002229 osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16]
   -88> 2014-11-07 16:14:57.006107 7f8608648700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.006107, event: reached_pg, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -87> 2014-11-07 16:14:57.006171 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] handle_message: 0x28b3a800
   -86> 2014-11-07 16:14:57.006237 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify trans b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 v 107796'32423868 (transaction) 176
   -85> 2014-11-07 16:14:57.006254 7f8608648700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.006254, event: started, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -84> 2014-11-07 16:14:57.006278 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423867 lc 107786'32421641 (107765'32418540,107796'32423867] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log log((107765'32418540,107796'32423867], crt=107792'32423690) [107796'32423868 (107796'32423845) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681747 2014-11-07 16:14:56.986326]
   -83> 2014-11-07 16:14:57.006309 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] add_log_entry 107796'32423868 (107796'32423845) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681747 2014-11-07 16:14:56.986326
   -82> 2014-11-07 16:14:57.006344 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log  adding 1 keys
   -81> 2014-11-07 16:14:57.006361 7f8608648700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log: trimming to 107786'32421641 entries 
   -80> 2014-11-07 16:14:57.006406 7f8608648700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'32423868, trimmed: 
   -79> 2014-11-07 16:14:57.006429 7f8608648700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.006429, event: started, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -78> 2014-11-07 16:14:57.006451 7f8608648700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.456 0x7576250)/0x7576250
   -77> 2014-11-07 16:14:57.006459 7f8608648700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724756 0x1eb3d0f8
   -76> 2014-11-07 16:14:57.006470 7f8608648700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.006470, event: commit_queued_for_journal_write, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -75> 2014-11-07 16:14:57.006500 7f8608648700 15 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] do_sub_op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11
   -74> 2014-11-07 16:14:57.006519 7f8608648700 10 osd.21 107796 dequeue_op 0x28b3a800 finish
   -73> 2014-11-07 16:14:57.006664 7f861cbdb700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.006664, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -72> 2014-11-07 16:14:57.007387 7f861c3da700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.007387, event: journaled_completion_queued, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -71> 2014-11-07 16:14:57.007400 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x1a05e140 seq 5724756 osr(6.456 0x7576250) 0x1eb3d0f8
   -70> 2014-11-07 16:14:57.007412 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x1a05e140 seq 5724756 osr(6.456 0x7576250) 346424 bytes   (queue has 1 ops and 346424 bytes)
   -69> 2014-11-07 16:14:57.007429 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x214251c0
   -68> 2014-11-07 16:14:57.007450 7f8619bd5700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x1a05e140 seq 5724756 osr(6.456 0x7576250)/0x7576250 start
   -67> 2014-11-07 16:14:57.007461 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x1eb3d0f8
   -66> 2014-11-07 16:14:57.007470 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b81fa/pglog_6.456/0//-1
   -65> 2014-11-07 16:14:57.007460 7f86183d2700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.007459, event: commit_sent, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -64> 2014-11-07 16:14:57.007497 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify_commit on op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.13
   -63> 2014-11-07 16:14:57.007538 7f86183d2700 20 osd.21 107796 share_map_peer 0x1fe6a840 already has epoch 107796
   -62> 2014-11-07 16:14:57.007542 7f86183d2700  1 -- 172.30.5.2:6803/25692 --> 172.30.5.1:6839/22281 -- osd_sub_op_reply(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] ondisk, result = 0) v2 -- ?+0 0x20f3c840 con 0x1fe6a840
   -61> 2014-11-07 16:14:57.007563 7f8619bd5700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724756.0.0
   -60> 2014-11-07 16:14:57.007567 7f8619bd5700 10 filestore  > header.spos 0.0.0
   -59> 2014-11-07 16:14:57.007651 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1
   -58> 2014-11-07 16:14:57.007684 7f8619bd5700 10 filestore oid: 16ef7597/infos/head//-1 not skipping op, *spos 5724756.0.1
   -57> 2014-11-07 16:14:57.007687 7f8619bd5700 10 filestore  > header.spos 0.0.0
   -56> 2014-11-07 16:14:57.007724 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_rmkeys meta/6b81fa/pglog_6.456/0//-1
   -55> 2014-11-07 16:14:57.007785 7f8619bd5700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724756.0.2
   -54> 2014-11-07 16:14:57.007791 7f8619bd5700 10 filestore  > header.spos 0.0.0
   -53> 2014-11-07 16:14:57.007812 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b81fa/pglog_6.456/0//-1
   -52> 2014-11-07 16:14:57.007842 7f8619bd5700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724756.0.3
   -51> 2014-11-07 16:14:57.007845 7f8619bd5700 10 filestore  > header.spos 0.0.0
   -50> 2014-11-07 16:14:57.007928 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 object_size 4194304 write_size 4194304
   -49> 2014-11-07 16:14:57.008028 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint hint 1048576 ret -95
   -48> 2014-11-07 16:14:57.008033 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) set_alloc_hint 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 object_size 4194304 write_size 4194304 = -95
   -47> 2014-11-07 16:14:57.008042 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) write 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 0~344064
   -46> 2014-11-07 16:14:57.008312 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) write 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 0~344064 = 344064
   -45> 2014-11-07 16:14:57.008349 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting '_'
   -44> 2014-11-07 16:14:57.008391 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting 'snapset'
   -43> 2014-11-07 16:14:57.008398 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6
   -42> 2014-11-07 16:14:57.008431 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 = 0
   -41> 2014-11-07 16:14:57.008454 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting '_'
   -40> 2014-11-07 16:14:57.008462 7f8619bd5700 20 filestore(/var/lib/ceph/osd/cloud-21) fgetattrs 652 getting 'snapset'
   -39> 2014-11-07 16:14:57.008466 7f8619bd5700 15 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6
   -38> 2014-11-07 16:14:57.008530 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) setattrs 6.456_head/b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 = 0
   -37> 2014-11-07 16:14:57.008543 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x1a05e140 seq 5724756 r = 0, finisher 0x255c0b80 0
   -36> 2014-11-07 16:14:57.008554 7f8619bd5700 10 filestore(/var/lib/ceph/osd/cloud-21) _finish_op 0x1a05e140 seq 5724756 osr(6.456 0x7576250)/0x7576250
   -35> 2014-11-07 16:14:57.008594 7f8618bd3700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.008594, event: sub_op_applied, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -34> 2014-11-07 16:14:57.008614 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify_applied on 0x1eb3d000 op osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[]) v11
   -33> 2014-11-07 16:14:57.008636 7f8618bd3700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] op_applied version 107796'32423868
   -32> 2014-11-07 16:14:57.008668 7f8618bd3700  5 -- op tracker -- seq: 11702, time: 2014-11-07 16:14:57.008668, event: done, op: osd_sub_op(client.7912580.0:24681747 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423868 snapset=0=[]:[] snapc=0=[])
   -31> 2014-11-07 16:14:57.021968 7f860d652700  1 -- 172.30.5.2:6804/25692 <== osd.77 172.30.5.6:0/32590 48 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:57.020796) v2 ==== 47+0+0 (1385461384 0 0) 0x26468400 con 0x2074a2c0
   -30> 2014-11-07 16:14:57.021988 7f860ee55700  1 -- 172.30.5.2:6805/25692 <== osd.77 172.30.5.6:0/32590 48 ==== osd_ping(ping e107796 stamp 2014-11-07 16:14:57.020796) v2 ==== 47+0+0 (1385461384 0 0) 0x21d18a00 con 0x204bf000
   -29> 2014-11-07 16:14:57.022025 7f860d652700  1 -- 172.30.5.2:6804/25692 --> 172.30.5.6:0/32590 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:57.020796) v2 -- ?+0 0x28a6a000 con 0x2074a2c0
   -28> 2014-11-07 16:14:57.022063 7f860d652700 20 osd.21 107796 share_map_peer 0x1fe01580 already has epoch 107796
   -27> 2014-11-07 16:14:57.022088 7f860ee55700  1 -- 172.30.5.2:6805/25692 --> 172.30.5.6:0/32590 -- osd_ping(ping_reply e107796 stamp 2014-11-07 16:14:57.020796) v2 -- ?+0 0x213b6400 con 0x204bf000
   -26> 2014-11-07 16:14:57.022164 7f860ee55700 20 osd.21 107796 share_map_peer 0x1fe01580 already has epoch 107796
   -25> 2014-11-07 16:14:57.022237 7f85f61a9700  1 -- 172.30.5.2:6803/25692 <== osd.13 172.30.5.1:6839/22281 213 ==== osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11 ==== 1404+0+1049401 (252162102 0 3108292305) 0x20eb8800 con 0x1fe6a840
   -24> 2014-11-07 16:14:57.022286 7f85f61a9700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.017183, event: header_read, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
   -23> 2014-11-07 16:14:57.022311 7f85f61a9700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.017185, event: throttled, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
   -22> 2014-11-07 16:14:57.022325 7f85f61a9700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.022220, event: all_read, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
   -21> 2014-11-07 16:14:57.022342 7f85f61a9700  5 -- op tracker -- seq: 11703, time: 0.000000, event: dispatched, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
   -20> 2014-11-07 16:14:57.022355 7f85f61a9700 10 osd.21 107796 handle_replica_op osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11 epoch 107796
   -19> 2014-11-07 16:14:57.022368 7f85f61a9700 20 osd.21 107796 should_share_map osd.13 172.30.5.1:6839/22281 107796
   -18> 2014-11-07 16:14:57.022379 7f85f61a9700 15 osd.21 107796 enqueue_op 0x21fa0e00 prio 127 cost 1049401 latency 0.005196 osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11
   -17> 2014-11-07 16:14:57.022427 7f8605e43700 10 osd.21 107796 dequeue_op 0x21fa0e00 prio 127 cost 1049401 latency 0.005243 osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11 pg pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16]
   -16> 2014-11-07 16:14:57.022464 7f8605e43700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.022464, event: reached_pg, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
   -15> 2014-11-07 16:14:57.022488 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] handle_message: 0x21fa0e00
   -14> 2014-11-07 16:14:57.022509 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify trans b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 v 107796'32423869 (transaction) 176
   -13> 2014-11-07 16:14:57.022526 7f8605e43700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.022526, event: started, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
   -12> 2014-11-07 16:14:57.022546 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423868 lc 107786'32421641 (107765'32418540,107796'32423868] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log log((107765'32418540,107796'32423868], crt=107792'32423690) [107796'32423869 (107796'32423868) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681748 2014-11-07 16:14:56.986423]
   -11> 2014-11-07 16:14:57.022584 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] add_log_entry 107796'32423869 (107796'32423868) modify   b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 by client.7912580.0:24681748 2014-11-07 16:14:56.986423
   -10> 2014-11-07 16:14:57.022617 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log  adding 1 keys
    -9> 2014-11-07 16:14:57.022633 7f8605e43700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] append_log: trimming to 107786'32421641 entries 
    -8> 2014-11-07 16:14:57.022673 7f8605e43700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 107796'32423869, trimmed: 
    -7> 2014-11-07 16:14:57.022702 7f8605e43700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.022702, event: started, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
    -6> 2014-11-07 16:14:57.022738 7f8605e43700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions existing osr(6.456 0x7576250)/0x7576250
    -5> 2014-11-07 16:14:57.022749 7f8605e43700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_transactions (writeahead) 5724757 0x1595fb78
    -4> 2014-11-07 16:14:57.022761 7f8605e43700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.022760, event: commit_queued_for_journal_write, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
    -3> 2014-11-07 16:14:57.022796 7f8605e43700 15 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] do_sub_op osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11
    -2> 2014-11-07 16:14:57.022847 7f8605e43700 10 osd.21 107796 dequeue_op 0x21fa0e00 finish
    -1> 2014-11-07 16:14:57.022940 7f861cbdb700  5 -- op tracker -- seq: 11703, time: 2014-11-07 16:14:57.022940, event: write_thread_in_journal_buffer, op: osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[])
     0> 2014-11-07 16:14:57.024605 7f8602e3d700 -1 *** Caught signal (Aborted) **
 in thread 7f8602e3d700

 ceph version 0.87-6-gdba7def (dba7defc623474ad17263c9fccfec60fe7a439f0)
 1: /usr/bin/ceph-osd() [0x9b6725]
 2: (()+0xfcb0) [0x7f8626439cb0]
 3: (gsignal()+0x35) [0x7f8624d3e0d5]
 4: (abort()+0x17b) [0x7f8624d4183b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f862569069d]
 6: (()+0xb5846) [0x7f862568e846]
 7: (()+0xb5873) [0x7f862568e873]
 8: (()+0xb596e) [0x7f862568e96e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xaa0089]
 10: (ReplicatedPG::trim_object(hobject_t const&)+0x222d) [0x8139ed]
 11: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim const&)+0x43e) [0x82b9be]
 12: (boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xc0) [0x870ce0]
 13: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_queued_events()+0xfb) [0x85618b]
 14: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, ReplicatedPG::NotTrimming, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x1e) [0x85633e]
 15: (ReplicatedPG::snap_trimmer()+0x4f8) [0x7d5ef8]
 16: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x673ab4]
 17: (ThreadPool::worker(ThreadPool::WorkThread*)+0x48e) [0xa8fade]
 18: (ThreadPool::WorkThread::entry()+0x10) [0xa92870]
 19: (()+0x7e9a) [0x7f8626431e9a]
 20: (clone()+0x6d) [0x7f8624dfc31d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
  20/20 osd
   0/ 5 optracker
   0/ 5 objclass
  20/20 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   1/ 1 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.21.log
--- end dump of recent events ---
2014-11-07 16:14:57.025606 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) _journaled_ahead 0x6298c30 seq 5724757 osr(6.456 0x7576250) 0x1595fb78
2014-11-07 16:14:57.025673 7f861bbd9700  5 filestore(/var/lib/ceph/osd/cloud-21) queue_op 0x6298c30 seq 5724757 osr(6.456 0x7576250) 1050936 bytes   (queue has 1 ops and 1050936 bytes)
2014-11-07 16:14:57.025695 7f861bbd9700 10 filestore(/var/lib/ceph/osd/cloud-21)  queueing ondisk 0x21424d00
2014-11-07 16:14:57.025715 7f86193d4700  5 filestore(/var/lib/ceph/osd/cloud-21) _do_op 0x6298c30 seq 5724757 osr(6.456 0x7576250)/0x7576250 start
2014-11-07 16:14:57.025725 7f86193d4700 10 filestore(/var/lib/ceph/osd/cloud-21) _do_transaction on 0x1595fb78
2014-11-07 16:14:57.025733 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/6b81fa/pglog_6.456/0//-1
2014-11-07 16:14:57.025786 7f86193d4700 10 filestore oid: 6b81fa/pglog_6.456/0//-1 not skipping op, *spos 5724757.0.0
2014-11-07 16:14:57.025791 7f86193d4700 10 filestore  > header.spos 0.0.0
2014-11-07 16:14:57.025774 7f86183d2700 10 osd.21 pg_epoch: 107796 pg[6.456( v 107796'32423869 lc 107786'32421641 (107765'32418540,107796'32423869] local-les=107794 n=9091 ec=164 les/c 107794/107791 107793/107793/104804) [13,105,21] r=2 lpr=107793 pi=106418-107792/48 luod=0'0 crt=107792'32423690 active m=16] sub_op_modify_commit on op osd_sub_op(client.7912580.0:24681748 6.456 b2dc7c56/rbd_data.74ae9c3be03aff.00000000000010b8/head//6 [] v 107796'32423869 snapset=0=[]:[] snapc=0=[]) v11, sending commit to osd.13
2014-11-07 16:14:57.025835 7f86193d4700 15 filestore(/var/lib/ceph/osd/cloud-21) _omap_setkeys meta/16ef7597/infos/head//-1

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com