Hi, I have some OSD's that keep committing suicide. My cluster has ~1.3M misplaced objects, and it can't really recover, because OSD's keep failing before recovering finishes. The load on the hosts is quite high, but the cluster currently has no other tasks than just the backfilling/recovering. I attached the logfile from a failed OSD. It shows the suicide, the recent events and also me starting the OSD again after some time. It'll keep running for a couple of hours and then fail again, for the same reason. I noticed a lot of timeouts. Apparently ceph stresses the hosts to the limit with the recovery tasks, so much that they timeout and can't finish that task. I don't understand why. Can I somehow throttle ceph a bit so that it doesn't keep overrunning itself? I kinda feel like it should chill out a bit and simply recover one step at a time instead of full force and then fail. Thanks, Erik.
2014-11-08 18:15:25.687596 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:05.687593) 2014-11-08 18:15:27.045265 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:07.045263) 2014-11-08 18:15:27.061317 7fc5aa30a700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:07.061315) 2014-11-08 18:15:28.049496 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:08.049493) 2014-11-08 18:15:28.163529 7fc5aa30a700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:08.163527) 2014-11-08 18:15:29.049618 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:09.049615) 2014-11-08 18:15:30.049760 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:10.049757) 2014-11-08 18:15:31.049920 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:11.049917) 2014-11-08 18:15:32.050078 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:12.050075) 2014-11-08 18:15:32.264214 7fc5aa30a700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:12.264212) 2014-11-08 18:15:33.050197 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:13.050195) 2014-11-08 18:15:33.884525 7fc5a39d2700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6809/5031236 pipe(0x1cd62680 sd=69 :50049 s=2 pgs=8208 cs=1 l=0 c=0x21468c60).fault with nothing to send, going to standby 2014-11-08 18:15:34.034198 7fc5a27c0700 0 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x1d22b440 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1e50c200).fault 2014-11-08 18:15:34.034307 7fc5a28c1700 0 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x23bd7900 sd=83 :0 s=1 pgs=0 cs=0 l=1 c=0x1cae0840).fault 2014-11-08 18:16:00.758335 7fc5a6cfe700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6806/1411 pipe(0x4379080 sd=29 :33012 s=2 pgs=3008 cs=1 l=0 c=0x1c5c2680).fault with nothing to send, going to standby 2014-11-08 18:16:00.759068 7fc5a56ef700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6806/12001550 pipe(0x22c1b700 sd=128 :6811 s=2 pgs=65553 cs=287 l=0 c=0x1e50aec0).fault with nothing to send, going to standby 2014-11-08 18:16:00.760390 7fc5a0c26700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6801/1238 pipe(0x226c0580 sd=190 :6811 s=2 pgs=1029 cs=3 l=0 c=0x1cec3b20).fault with nothing to send, going to standby 2014-11-08 18:16:00.767012 7fc5a152e700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6811/1606 pipe(0x23e1bc80 sd=37 :53443 s=2 pgs=3797 cs=9 l=0 c=0x1c5c0580).fault with nothing to send, going to standby 2014-11-08 18:16:00.767056 7fc5a61fa700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6802/1404 pipe(0x4379600 sd=43 :37320 s=2 pgs=7745 cs=7 l=0 c=0x1c7911e0).fault with nothing to send, going to standby 2014-11-08 18:16:02.183747 7fc5cd36a700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.17:6806/32391 pipe(0x224ee880 sd=191 :6811 s=2 pgs=774 cs=5 l=0 c=0x1cec4620).fault with nothing to send, going to standby 2014-11-08 18:16:02.270284 7fc5c746a700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6801/1266 pipe(0x2342f900 sd=194 :6811 s=2 pgs=64 cs=5 l=0 c=0x1cec4780).fault with nothing to send, going to standby 2014-11-08 18:16:02.275838 7fc5a0d27700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6807/3588 pipe(0x226c02c0 sd=184 :6811 s=2 pgs=208 cs=3 l=0 c=0x1cec4200).fault with nothing to send, going to standby 2014-11-08 18:16:02.985549 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:07.985682 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:12.985790 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:12.985817 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:17.985897 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:17.985918 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:22.985984 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:22.986006 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:27.986083 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:27.986099 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:32.986340 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:32.986359 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:37.986429 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:37.986445 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:42.986521 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:42.986551 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:47.986631 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:47.986644 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:52.986716 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:52.986740 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:16:57.986812 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:16:57.986833 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:02.986915 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:02.986939 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:07.987010 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:07.987045 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:12.987117 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:12.987134 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:17.987205 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:17.987233 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:22.987303 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:22.987323 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:27.987395 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:27.987420 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:32.987496 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:32.987511 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:37.987585 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:37.987615 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:42.987684 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:42.987707 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:47.987784 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:47.987832 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:52.987905 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:52.987924 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:17:57.987996 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:17:57.988018 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:18:02.988091 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:18:02.988109 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:18:07.988249 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:18:07.988270 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:18:12.988344 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:18:12.988368 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:18:17.988440 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 2014-11-08 18:18:17.988466 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 2014-11-08 18:18:17.988468 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had suicide timed out after 150 2014-11-08 18:18:18.249885 7fc5c9375700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fc5c9375700 time 2014-11-08 18:18:18.109404 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xab6905] 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0x9ef1a9] 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0x9efa9e] 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x9f01bc] 5: (CephContextServiceThread::entry()+0x15b) [0xacb3fb] 6: (()+0x7ee5) [0x7fc5cc365ee5] 7: (clone()+0x6d) [0x7fc5cb04bb8d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -9999> 2014-11-08 18:14:21.940960 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3713 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:21.938927) v2 ==== 47+0+0 (2240051367 0 0) 0x1e891a40 con 0x1cae0c60 -9998> 2014-11-08 18:14:21.941028 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3713 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:21.938927) v2 ==== 47+0+0 (2240051367 0 0) 0x227b6720 con 0x1c5c02c0 -9997> 2014-11-08 18:14:21.941041 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3713 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:21.938927) v2 ==== 47+0+0 (2240051367 0 0) 0x246cd820 con 0x1cae0160 -9996> 2014-11-08 18:14:21.970907 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59213, attrs_read=0)) v1 -- ?+0 0x224bd000 con 0x21468c60 -9995> 2014-11-08 18:14:21.970954 7fc5af314700 5 -- op tracker -- seq: 46653, time: 2014-11-08 18:14:21.970953, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59213, to_read={e21d314e/1000002b717.00000035/head//4=0,4194304}, attrs_to_read=)) -9994> 2014-11-08 18:14:22.395780 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1decfda0 con 0x1e50aec0 -9993> 2014-11-08 18:14:22.395811 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(88e990b7/1000001acc4.00000011/head//4)]) v2 -- ?+0 0x1d5b5200 con 0x1e50aec0 -9992> 2014-11-08 18:14:22.395986 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1decdfa0 con 0x21468c60 -9991> 2014-11-08 18:14:22.395998 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1decb660 con 0x21468c60 -9990> 2014-11-08 18:14:22.396005 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1decd460 con 0x21468c60 -9989> 2014-11-08 18:14:22.396010 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1decc560 con 0x21468c60 -9988> 2014-11-08 18:14:22.396016 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(b150bddd/1000002a5c5.00000094/head//4),PushReplyOp(b1f0bddd/1000001a92b.00000008/head//4),PushReplyOp(4731bddd/1000001957b.00000000/head//4),PushReplyOp(ae22bddd/10000028e7c.0000001a/head//4)]) v2 -- ?+0 0x1d5b5e00 con 0x21468c60 -9987> 2014-11-08 18:14:22.434812 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1555 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) v3 ==== 757+0+0 (856205052 0 0) 0x1d650480 con 0x21468c60 -9986> 2014-11-08 18:14:22.435283 7fc5a39d2700 5 -- op tracker -- seq: 46655, time: 2014-11-08 18:14:22.434725, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9985> 2014-11-08 18:14:22.435291 7fc5a39d2700 5 -- op tracker -- seq: 46655, time: 2014-11-08 18:14:22.434728, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9984> 2014-11-08 18:14:22.435294 7fc5a39d2700 5 -- op tracker -- seq: 46655, time: 2014-11-08 18:14:22.434805, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9983> 2014-11-08 18:14:22.435297 7fc5a39d2700 5 -- op tracker -- seq: 46655, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9982> 2014-11-08 18:14:22.435324 7fc5acb0f700 5 -- op tracker -- seq: 46655, time: 2014-11-08 18:14:22.435324, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9981> 2014-11-08 18:14:22.435331 7fc5acb0f700 5 -- op tracker -- seq: 46655, time: 2014-11-08 18:14:22.435331, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9980> 2014-11-08 18:14:22.435377 7fc5acb0f700 5 -- op tracker -- seq: 46655, time: 2014-11-08 18:14:22.435377, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb ae22bddd/10000028e7c.0000001a/head//4) -9979> 2014-11-08 18:14:22.535778 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1497 ==== pg_stats_ack(4 pgs tid 1322) v1 ==== 120+0+0 (3697705507 0 0) 0x231410e0 con 0x44ed6a0 -9978> 2014-11-08 18:14:22.591441 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 488 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:22.589362) v2 ==== 47+0+0 (3521727356 0 0) 0x1d0712c0 con 0x1eb9d800 -9977> 2014-11-08 18:14:22.591506 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:22.589362) v2 -- ?+0 0x1d5e5820 con 0x1eb9d800 -9976> 2014-11-08 18:14:22.592092 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 488 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:22.589362) v2 ==== 47+0+0 (3521727356 0 0) 0x22d84560 con 0x1eb9c8e0 -9975> 2014-11-08 18:14:22.592110 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:22.589362) v2 -- ?+0 0x2230ba20 con 0x1eb9c8e0 -9974> 2014-11-08 18:14:22.602315 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x1deccce0 con 0x21468c60 -9973> 2014-11-08 18:14:22.602342 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.8fs0 5773 [PushReplyOp(a407678f/1000002a70f.0000001f/head//4)]) v2 -- ?+0 0x1ebb4200 con 0x21468c60 -9972> 2014-11-08 18:14:22.602356 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1ce16ae0 con 0x1c5c0580 -9971> 2014-11-08 18:14:22.602366 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1ce17440 con 0x1c5c0580 -9970> 2014-11-08 18:14:22.602373 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1ce141a0 con 0x1c5c0580 -9969> 2014-11-08 18:14:22.602379 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1ce103c0 con 0x1c5c0580 -9968> 2014-11-08 18:14:22.602385 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d635be0 con 0x1c5c0580 -9967> 2014-11-08 18:14:22.602391 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(b1835b16/1000001b457.00000045/head//4),PushReplyOp(cb645b16/1000002a768.000002f8/head//4),PushReplyOp(d8845b16/1000001a0b9.0000045d/head//4),PushReplyOp(c6b45b16/100000196f7.0000004a/head//4),PushReplyOp(82e45b16/10000019d1f.00000043/head//4)]) v2 -- ?+0 0x2393dc00 con 0x1c5c0580 -9966> 2014-11-08 18:14:22.658436 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1556 ==== pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) v3 ==== 757+0+0 (1190183007 0 0) 0x1d651b00 con 0x21468c60 -9965> 2014-11-08 18:14:22.658471 7fc5a39d2700 5 -- op tracker -- seq: 46656, time: 2014-11-08 18:14:22.658340, event: header_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9964> 2014-11-08 18:14:22.658480 7fc5a39d2700 5 -- op tracker -- seq: 46656, time: 2014-11-08 18:14:22.658345, event: throttled, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9963> 2014-11-08 18:14:22.658483 7fc5a39d2700 5 -- op tracker -- seq: 46656, time: 2014-11-08 18:14:22.658426, event: all_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9962> 2014-11-08 18:14:22.658486 7fc5a39d2700 5 -- op tracker -- seq: 46656, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9961> 2014-11-08 18:14:22.658528 7fc5af314700 5 -- op tracker -- seq: 46656, time: 2014-11-08 18:14:22.658528, event: reached_pg, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9960> 2014-11-08 18:14:22.658538 7fc5af314700 5 -- op tracker -- seq: 46656, time: 2014-11-08 18:14:22.658538, event: started, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9959> 2014-11-08 18:14:22.658546 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1557 ==== MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (531779791 0 0) 0x2287a1c0 con 0x21468c60 -9958> 2014-11-08 18:14:22.658556 7fc5a39d2700 5 -- op tracker -- seq: 46657, time: 2014-11-08 18:14:22.658516, event: header_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) -9957> 2014-11-08 18:14:22.658560 7fc5a39d2700 5 -- op tracker -- seq: 46657, time: 2014-11-08 18:14:22.658517, event: throttled, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) -9956> 2014-11-08 18:14:22.658563 7fc5a39d2700 5 -- op tracker -- seq: 46657, time: 2014-11-08 18:14:22.658545, event: all_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) -9955> 2014-11-08 18:14:22.658567 7fc5a39d2700 5 -- op tracker -- seq: 46657, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) -9954> 2014-11-08 18:14:22.658586 7fc5af314700 5 -- op tracker -- seq: 46656, time: 2014-11-08 18:14:22.658586, event: done, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb a407678f/1000002a70f.0000001f/head//4) -9953> 2014-11-08 18:14:22.658594 7fc5acb0f700 5 -- op tracker -- seq: 46657, time: 2014-11-08 18:14:22.658593, event: reached_pg, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) -9952> 2014-11-08 18:14:22.666439 7fc5c362f700 5 osd.9 5773 tick -9951> 2014-11-08 18:14:22.871426 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8412 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) v3 ==== 757+0+0 (2490237011 0 0) 0x1e807080 con 0x1c5c0580 -9950> 2014-11-08 18:14:22.871463 7fc5a152e700 5 -- op tracker -- seq: 46658, time: 2014-11-08 18:14:22.871189, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9949> 2014-11-08 18:14:22.871473 7fc5a152e700 5 -- op tracker -- seq: 46658, time: 2014-11-08 18:14:22.871194, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9948> 2014-11-08 18:14:22.871479 7fc5a152e700 5 -- op tracker -- seq: 46658, time: 2014-11-08 18:14:22.871398, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9947> 2014-11-08 18:14:22.871482 7fc5a152e700 5 -- op tracker -- seq: 46658, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9946> 2014-11-08 18:14:22.871524 7fc5adb11700 5 -- op tracker -- seq: 46658, time: 2014-11-08 18:14:22.871524, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9945> 2014-11-08 18:14:22.871538 7fc5adb11700 5 -- op tracker -- seq: 46658, time: 2014-11-08 18:14:22.871537, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9944> 2014-11-08 18:14:22.871586 7fc5adb11700 5 -- op tracker -- seq: 46658, time: 2014-11-08 18:14:22.871586, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e1dfecb5/10000021872.00000000/head//4) -9943> 2014-11-08 18:14:22.874792 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8413 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) v3 ==== 757+0+0 (296140568 0 0) 0x1d869f80 con 0x1c5c0580 -9942> 2014-11-08 18:14:22.874834 7fc5a152e700 5 -- op tracker -- seq: 46659, time: 2014-11-08 18:14:22.874735, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9941> 2014-11-08 18:14:22.874841 7fc5a152e700 5 -- op tracker -- seq: 46659, time: 2014-11-08 18:14:22.874737, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9940> 2014-11-08 18:14:22.874844 7fc5a152e700 5 -- op tracker -- seq: 46659, time: 2014-11-08 18:14:22.874788, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9939> 2014-11-08 18:14:22.874849 7fc5a152e700 5 -- op tracker -- seq: 46659, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9938> 2014-11-08 18:14:22.874891 7fc5ad310700 5 -- op tracker -- seq: 46659, time: 2014-11-08 18:14:22.874891, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9937> 2014-11-08 18:14:22.874900 7fc5ad310700 5 -- op tracker -- seq: 46659, time: 2014-11-08 18:14:22.874900, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9936> 2014-11-08 18:14:22.874932 7fc5ad310700 5 -- op tracker -- seq: 46659, time: 2014-11-08 18:14:22.874932, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 82e45b16/10000019d1f.00000043/head//4) -9935> 2014-11-08 18:14:22.906080 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8414 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) v2 ==== 110+0+0 (3901745953 0 0) 0x2393dc00 con 0x1c5c0580 -9934> 2014-11-08 18:14:22.906110 7fc5a152e700 5 -- op tracker -- seq: 46660, time: 2014-11-08 18:14:22.905991, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9933> 2014-11-08 18:14:22.906120 7fc5a152e700 5 -- op tracker -- seq: 46660, time: 2014-11-08 18:14:22.905994, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9932> 2014-11-08 18:14:22.906125 7fc5a152e700 5 -- op tracker -- seq: 46660, time: 2014-11-08 18:14:22.906070, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9931> 2014-11-08 18:14:22.906130 7fc5a152e700 5 -- op tracker -- seq: 46660, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9930> 2014-11-08 18:14:22.906168 7fc5aeb13700 5 -- op tracker -- seq: 46660, time: 2014-11-08 18:14:22.906168, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9929> 2014-11-08 18:14:22.906212 7fc5aeb13700 5 -- op tracker -- seq: 46660, time: 2014-11-08 18:14:22.906212, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9928> 2014-11-08 18:14:23.062803 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8415 ==== MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (851421737 0 0) 0x229bcc00 con 0x1c5c0580 -9927> 2014-11-08 18:14:23.062846 7fc5a152e700 5 -- op tracker -- seq: 46661, time: 2014-11-08 18:14:23.035605, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9926> 2014-11-08 18:14:23.062862 7fc5a152e700 5 -- op tracker -- seq: 46661, time: 2014-11-08 18:14:23.035610, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9925> 2014-11-08 18:14:23.062875 7fc5a152e700 5 -- op tracker -- seq: 46661, time: 2014-11-08 18:14:23.062781, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9924> 2014-11-08 18:14:23.062888 7fc5a152e700 5 -- op tracker -- seq: 46661, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9923> 2014-11-08 18:14:23.062928 7fc5b0316700 5 -- op tracker -- seq: 46661, time: 2014-11-08 18:14:23.062928, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9922> 2014-11-08 18:14:23.063006 7fc5a152e700 5 -- op tracker -- seq: 46661, time: 2014-11-08 18:14:23.063006, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(6b50fcb5/1000001a6ca.0000003c/head//4, version: 397'2197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6b50fcb5/1000001a6ca.0000003c/head//4@397'2197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9921> 2014-11-08 18:14:23.114934 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.8fs0 5773 ECSubReadReply(tid=59215, attrs_read=0)) v1 -- ?+0 0x21838500 con 0x21468c60 -9920> 2014-11-08 18:14:23.117796 7fc5acb0f700 5 -- op tracker -- seq: 46657, time: 2014-11-08 18:14:23.117795, event: done, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59215, to_read={7d27678f/1000001a107.0000002f/head//4=0,4194304}, attrs_to_read=)) -9919> 2014-11-08 18:14:23.140296 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22e9b0c0 con 0x1c5c0580 -9918> 2014-11-08 18:14:23.168409 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(6b50fcb5/1000001a6ca.0000003c/head//4)]) v2 -- ?+0 0x229b8a00 con 0x1c5c0580 -9917> 2014-11-08 18:14:23.195031 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1558 ==== MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (880906665 0 0) 0x1ebb4200 con 0x21468c60 -9916> 2014-11-08 18:14:23.195100 7fc5a39d2700 5 -- op tracker -- seq: 46662, time: 2014-11-08 18:14:23.154091, event: header_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9915> 2014-11-08 18:14:23.195120 7fc5a39d2700 5 -- op tracker -- seq: 46662, time: 2014-11-08 18:14:23.154095, event: throttled, op: MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9914> 2014-11-08 18:14:23.195133 7fc5a39d2700 5 -- op tracker -- seq: 46662, time: 2014-11-08 18:14:23.195019, event: all_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9913> 2014-11-08 18:14:23.195145 7fc5a39d2700 5 -- op tracker -- seq: 46662, time: 0.000000, event: dispatched, op: MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9912> 2014-11-08 18:14:23.195191 7fc5af314700 5 -- op tracker -- seq: 46662, time: 2014-11-08 18:14:23.195191, event: reached_pg, op: MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9911> 2014-11-08 18:14:23.197028 7fc5a39d2700 5 -- op tracker -- seq: 46662, time: 2014-11-08 18:14:23.197028, event: done, op: MOSDPGPush(4.8fs2 5773 [PushOp(7d27678f/1000001a107.0000002f/head//4, version: 397'1656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7d27678f/1000001a107.0000002f/head//4@397'1656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9910> 2014-11-08 18:14:23.240225 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8416 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) v2 ==== 370+0+0 (3946142085 0 0) 0x229b8a00 con 0x1c5c0580 -9909> 2014-11-08 18:14:23.240246 7fc5a152e700 5 -- op tracker -- seq: 46663, time: 2014-11-08 18:14:23.240125, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) -9908> 2014-11-08 18:14:23.240253 7fc5a152e700 5 -- op tracker -- seq: 46663, time: 2014-11-08 18:14:23.240127, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) -9907> 2014-11-08 18:14:23.240258 7fc5a152e700 5 -- op tracker -- seq: 46663, time: 2014-11-08 18:14:23.240220, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) -9906> 2014-11-08 18:14:23.240262 7fc5a152e700 5 -- op tracker -- seq: 46663, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) -9905> 2014-11-08 18:14:23.240300 7fc5afb15700 5 -- op tracker -- seq: 46663, time: 2014-11-08 18:14:23.240300, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) -9904> 2014-11-08 18:14:23.240409 7fc5afb15700 5 -- op tracker -- seq: 46663, time: 2014-11-08 18:14:23.240409, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(6932e343/1000001a9b0.0000012e/head//4),PushReplyOp(36d2e343/10000019d56.00000060/head//4),PushReplyOp(aad3e343/1000001a890.00000017/head//4),PushReplyOp(6c04e343/10000017a6f.00000000/head//4),PushReplyOp(ab24e343/10000019b71.0000004b/head//4)]) -9903> 2014-11-08 18:14:23.241080 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=7546e343/1000002a345.0000003b/head//4,576e343/1000001a1db.000002f0/head//4,1ff6e343/1000002b8dd.0000007e/head//4,7348e343/1000002a3ac.00000012/head//4,7648e343/1000002b75e.0000006e/head//4)) v1 -- ?+0 0x25629680 con 0x1e50aec0 -9902> 2014-11-08 18:14:23.241104 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x25629d40 con 0x42c86e0 -9901> 2014-11-08 18:14:23.241119 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=5 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 7fd8e343/10000028cf9.00000005/head//4 -9900> 2014-11-08 18:14:23.241147 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb ab24e343/10000019b71.0000004b/head//4) v3 -- ?+0 0x1d654380 con 0x1c5c0580 -9899> 2014-11-08 18:14:23.241242 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x25629d40 con 0x42c86e0 -9898> 2014-11-08 18:14:23.241256 7fc5b9328700 5 -- op tracker -- seq: 46664, time: 2014-11-08 18:14:23.241113, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) -9897> 2014-11-08 18:14:23.241262 7fc5b9328700 5 -- op tracker -- seq: 46664, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) -9896> 2014-11-08 18:14:23.241267 7fc5b9328700 5 -- op tracker -- seq: 46664, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) -9895> 2014-11-08 18:14:23.241271 7fc5b9328700 5 -- op tracker -- seq: 46664, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) -9894> 2014-11-08 18:14:23.241298 7fc5ad310700 5 -- op tracker -- seq: 46664, time: 2014-11-08 18:14:23.241298, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) -9893> 2014-11-08 18:14:23.258695 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22e9c560 con 0x21468c60 -9892> 2014-11-08 18:14:23.258789 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.8fs0 5773 [PushReplyOp(7d27678f/1000001a107.0000002f/head//4)]) v2 -- ?+0 0x23938c00 con 0x21468c60 -9891> 2014-11-08 18:14:23.589090 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 310 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:23.586837) v2 ==== 47+0+0 (3898365483 0 0) 0x22e5f9e0 con 0x1eb9d6a0 -9890> 2014-11-08 18:14:23.589080 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 310 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:23.586837) v2 ==== 47+0+0 (3898365483 0 0) 0x2026d0a0 con 0x1eb9c780 -9889> 2014-11-08 18:14:23.589132 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:23.586837) v2 -- ?+0 0x22d84560 con 0x1eb9c780 -9888> 2014-11-08 18:14:23.589239 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:23.586837) v2 -- ?+0 0x1d0712c0 con 0x1eb9d6a0 -9887> 2014-11-08 18:14:23.666590 7fc5c362f700 5 osd.9 5773 tick -9886> 2014-11-08 18:14:23.727474 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1559 ==== MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 3316125+0+0 (473459252 0 0) 0x23938c00 con 0x21468c60 -9885> 2014-11-08 18:14:23.727541 7fc5a39d2700 5 -- op tracker -- seq: 46665, time: 2014-11-08 18:14:23.660809, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9884> 2014-11-08 18:14:23.727573 7fc5a39d2700 5 -- op tracker -- seq: 46665, time: 2014-11-08 18:14:23.660814, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9883> 2014-11-08 18:14:23.727596 7fc5a39d2700 5 -- op tracker -- seq: 46665, time: 2014-11-08 18:14:23.727439, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9882> 2014-11-08 18:14:23.727617 7fc5a39d2700 5 -- op tracker -- seq: 46665, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9881> 2014-11-08 18:14:23.727632 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) v1 -- ?+0 0x211d6e00 con 0x42c86e0 -9880> 2014-11-08 18:14:23.727652 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d6e00 con 0x42c86e0 -9879> 2014-11-08 18:14:23.727655 7fc5ad310700 5 -- op tracker -- seq: 46664, time: 2014-11-08 18:14:23.727655, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4382, to_read={7546e343/1000002a345.0000003b/head//4=0,4194304,576e343/1000001a1db.000002f0/head//4=0,4194304,1ff6e343/1000002b8dd.0000007e/head//4=0,4194304,7348e343/1000002a3ac.00000012/head//4=0,4194304,7648e343/1000002b75e.0000006e/head//4=0,4194304}, attrs_to_read=)) -9878> 2014-11-08 18:14:23.727662 7fc5b9328700 5 -- op tracker -- seq: 46666, time: 2014-11-08 18:14:23.727645, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) -9877> 2014-11-08 18:14:23.727666 7fc5b9328700 5 -- op tracker -- seq: 46666, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) -9876> 2014-11-08 18:14:23.727668 7fc5b9328700 5 -- op tracker -- seq: 46666, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) -9875> 2014-11-08 18:14:23.727670 7fc5b9328700 5 -- op tracker -- seq: 46666, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) -9874> 2014-11-08 18:14:23.727714 7fc5afb15700 5 -- op tracker -- seq: 46666, time: 2014-11-08 18:14:23.727714, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) -9873> 2014-11-08 18:14:23.727758 7fc5afb15700 5 -- op tracker -- seq: 46666, time: 2014-11-08 18:14:23.727758, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=0)) -9872> 2014-11-08 18:14:23.727803 7fc5acb0f700 5 -- op tracker -- seq: 46665, time: 2014-11-08 18:14:23.727803, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9871> 2014-11-08 18:14:23.727939 7fc5acb0f700 5 -- op tracker -- seq: 46665, time: 2014-11-08 18:14:23.727938, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(b6c3bddd/10000021770.00000000/head//4, version: 3033'6795, data_included: [0~102400], data_size: 102400, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(b6c3bddd/10000021770.00000000/head//4@3033'6795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:204800, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e704bddd/10000016493.00000000/head//4, version: 397'1079, data_included: [0~1110016], data_size: 1110016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(e704bddd/10000016493.00000000/head//4@397'1079, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2220032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c714bddd/1000000ca62.00000000/head//4, version: 397'404, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c714bddd/1000000ca62.00000000/head//4@397'404, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df64bddd/100000296f3.00000095/head//4, version: 3033'8377, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df64bddd/100000296f3. 00000095/head//4@3033'8377, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9870> 2014-11-08 18:14:24.038758 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3698 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:24.036288) v2 ==== 47+0+0 (1184444484 0 0) 0x2026eae0 con 0x1c9f8580 -9869> 2014-11-08 18:14:24.038806 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:24.036288) v2 -- ?+0 0x22e5f9e0 con 0x1c9f8580 -9868> 2014-11-08 18:14:24.038829 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3698 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:24.036288) v2 ==== 47+0+0 (1184444484 0 0) 0x1db1d640 con 0x1c8b7d20 -9867> 2014-11-08 18:14:24.038839 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:24.036288) v2 -- ?+0 0x2026d0a0 con 0x1c8b7d20 -9866> 2014-11-08 18:14:24.304382 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2143 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) v1 ==== 10488220+0+0 (1049875301 0 0) 0x21a4bc00 con 0x1e50aec0 -9865> 2014-11-08 18:14:24.304417 7fc5a56ef700 5 -- op tracker -- seq: 46667, time: 2014-11-08 18:14:24.096399, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) -9864> 2014-11-08 18:14:24.304427 7fc5a56ef700 5 -- op tracker -- seq: 46667, time: 2014-11-08 18:14:24.096405, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) -9863> 2014-11-08 18:14:24.304429 7fc5a56ef700 5 -- op tracker -- seq: 46667, time: 2014-11-08 18:14:24.304372, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) -9862> 2014-11-08 18:14:24.304433 7fc5a56ef700 5 -- op tracker -- seq: 46667, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) -9861> 2014-11-08 18:14:24.304472 7fc5ad310700 5 -- op tracker -- seq: 46667, time: 2014-11-08 18:14:24.304472, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) -9860> 2014-11-08 18:14:24.324730 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x22e9f9e0 con 0x21468c60 -9859> 2014-11-08 18:14:24.324766 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x22e9eea0 con 0x21468c60 -9858> 2014-11-08 18:14:24.324773 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x22e99680 con 0x21468c60 -9857> 2014-11-08 18:14:24.324779 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x22d80000 con 0x21468c60 -9856> 2014-11-08 18:14:24.346608 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8417 ==== MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c865b16/ 1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (870921369 0 0) 0x229bcc00 con 0x1c5c0580 -9855> 2014-11-08 18:14:24.349723 7fc5a152e700 5 -- op tracker -- seq: 46668, time: 2014-11-08 18:14:24.159225, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c865b16/1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9854> 2014-11-08 18:14:24.349772 7fc5a152e700 5 -- op tracker -- seq: 46668, time: 2014-11-08 18:14:24.159231, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 5c865b16/1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9853> 2014-11-08 18:14:24.349797 7fc5a152e700 5 -- op tracker -- seq: 46668, time: 2014-11-08 18:14:24.346592, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c865b16/ 1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9852> 2014-11-08 18:14:24.349821 7fc5a152e700 5 -- op tracker -- seq: 46668, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c865b16/ 1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9851> 2014-11-08 18:14:24.349885 7fc5afb15700 5 -- op tracker -- seq: 46668, time: 2014-11-08 18:14:24.349885, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c865b16/1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9850> 2014-11-08 18:14:24.350079 7fc5afb15700 5 -- op tracker -- seq: 46668, time: 2014-11-08 18:14:24.350079, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(d5255b16/1000002bc82.0000012f/head//4, version: 3485'7636, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d5255b16/1000002bc82.0000012f/head//4@3485'7636, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7cd55b16/10000019b02.0000032d/head//4, version: 397'1015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7cd55b16/10000019b02.0000032d/head//4@397'1015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f165b16/1000002911d.00000002/head//4, version: 3033'4038, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f165b16/1000002911d.00000002/head//4@3033'4038, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c865b16/1000002a334.00000063/head//4, version: 3033'4817, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c865b16/ 1000002a334.00000063/head//4@3033'4817, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9b65b16/10000019f97.00000010/head//4, version: 397'1503, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9b65b16/10000019f97.00000010/head//4@397'1503, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9849> 2014-11-08 18:14:24.360851 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(7546e343/1000002a345.0000003b/head//4, version: 3033'9339, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7546e343/1000002a345.0000003b/head//4@3033'9339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(576e343/1000001a1db.000002f0/head//4, version: 397'3507, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(576e343/1000001a1db.000002f0/head//4@397'3507, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ff6e343/1000002b8dd.0000007e/head//4, version: 3473'13828, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ff6e343/1000002b8dd.0000007e/head//4@3473'13828, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7348e343/1000002a3ac.00000012/head//4, version: 3033'9409, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7348e343/1000002a3ac.00000012/head// 4@3033'9409, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7648e343/1000002b75e.0000006e/head//4, version: 3473'13525, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7648e343/1000002b75e.0000006e/head//4@3473'13525, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393b600 con 0x1c5c0580 -9848> 2014-11-08 18:14:24.360969 7fc5ad310700 5 -- op tracker -- seq: 46667, time: 2014-11-08 18:14:24.360969, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4382, attrs_read=5)) -9847> 2014-11-08 18:14:24.637336 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8418 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x1ce141a0 con 0x1c5c0580 -9846> 2014-11-08 18:14:24.637374 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637169, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9845> 2014-11-08 18:14:24.637385 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637173, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9844> 2014-11-08 18:14:24.637391 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637272, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9843> 2014-11-08 18:14:24.637396 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637369, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9842> 2014-11-08 18:14:24.637403 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637402, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9841> 2014-11-08 18:14:24.637413 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637413, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9840> 2014-11-08 18:14:24.637422 7fc5b9b29700 5 -- op tracker -- seq: 46669, time: 2014-11-08 18:14:24.637422, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9839> 2014-11-08 18:14:24.637435 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8419 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x1ce17440 con 0x1c5c0580 -9838> 2014-11-08 18:14:24.637445 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637290, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9837> 2014-11-08 18:14:24.637451 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637290, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9836> 2014-11-08 18:14:24.637456 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637312, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9835> 2014-11-08 18:14:24.637461 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637444, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9834> 2014-11-08 18:14:24.637467 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637467, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9833> 2014-11-08 18:14:24.637473 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637473, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9832> 2014-11-08 18:14:24.637479 7fc5b9b29700 5 -- op tracker -- seq: 46670, time: 2014-11-08 18:14:24.637479, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9831> 2014-11-08 18:14:24.637488 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8420 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x1ce16ae0 con 0x1c5c0580 -9830> 2014-11-08 18:14:24.637497 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637318, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9829> 2014-11-08 18:14:24.637503 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637319, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9828> 2014-11-08 18:14:24.637508 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637339, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9827> 2014-11-08 18:14:24.637513 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637496, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9826> 2014-11-08 18:14:24.637519 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637519, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9825> 2014-11-08 18:14:24.637525 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637525, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9824> 2014-11-08 18:14:24.637531 7fc5b9b29700 5 -- op tracker -- seq: 46671, time: 2014-11-08 18:14:24.637531, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9823> 2014-11-08 18:14:24.637507 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8423 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) v2 ==== 370+0+0 (2794589905 0 0) 0x2393b600 con 0x1c5c0580 -9822> 2014-11-08 18:14:24.637539 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8421 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x22a25460 con 0x1c5c0580 -9821> 2014-11-08 18:14:24.637547 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637345, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9820> 2014-11-08 18:14:24.637553 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637345, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9819> 2014-11-08 18:14:24.637556 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637366, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9818> 2014-11-08 18:14:24.637559 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637546, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9817> 2014-11-08 18:14:24.637564 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637564, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9816> 2014-11-08 18:14:24.637570 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637570, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9815> 2014-11-08 18:14:24.637551 7fc5a152e700 5 -- op tracker -- seq: 46673, time: 2014-11-08 18:14:24.637437, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) -9814> 2014-11-08 18:14:24.637579 7fc5b9b29700 5 -- op tracker -- seq: 46672, time: 2014-11-08 18:14:24.637579, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9813> 2014-11-08 18:14:24.637587 7fc5a152e700 5 -- op tracker -- seq: 46673, time: 2014-11-08 18:14:24.637437, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) -9812> 2014-11-08 18:14:24.637594 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8422 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x22a24560 con 0x1c5c0580 -9811> 2014-11-08 18:14:24.637610 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637380, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9810> 2014-11-08 18:14:24.637606 7fc5a152e700 5 -- op tracker -- seq: 46673, time: 2014-11-08 18:14:24.637506, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) -9809> 2014-11-08 18:14:24.637618 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637381, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9808> 2014-11-08 18:14:24.637628 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637431, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9807> 2014-11-08 18:14:24.637633 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637609, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9806> 2014-11-08 18:14:24.637624 7fc5a152e700 5 -- op tracker -- seq: 46673, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) -9805> 2014-11-08 18:14:24.637639 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637639, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9804> 2014-11-08 18:14:24.637651 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637651, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9803> 2014-11-08 18:14:24.637658 7fc5b9b29700 5 -- op tracker -- seq: 46674, time: 2014-11-08 18:14:24.637658, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9802> 2014-11-08 18:14:24.637689 7fc5afb15700 5 -- op tracker -- seq: 46673, time: 2014-11-08 18:14:24.637689, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) -9801> 2014-11-08 18:14:24.637818 7fc5afb15700 5 -- op tracker -- seq: 46673, time: 2014-11-08 18:14:24.637818, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7546e343/1000002a345.0000003b/head//4),PushReplyOp(576e343/1000001a1db.000002f0/head//4),PushReplyOp(1ff6e343/1000002b8dd.0000007e/head//4),PushReplyOp(7348e343/1000002a3ac.00000012/head//4),PushReplyOp(7648e343/1000002b75e.0000006e/head//4)]) -9800> 2014-11-08 18:14:24.638080 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=7fd8e343/10000028cf9.00000005/head//4)) v1 -- ?+0 0x2562bf00 con 0x1e50aec0 -9799> 2014-11-08 18:14:24.638099 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2562b600 con 0x42c86e0 -9798> 2014-11-08 18:14:24.638112 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is a969e343/1000002baa9.00000005/head//4 -9797> 2014-11-08 18:14:24.638136 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb 7648e343/1000002b75e.0000006e/head//4) v3 -- ?+0 0x1e800900 con 0x1c5c0580 -9796> 2014-11-08 18:14:24.638166 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2562b600 con 0x42c86e0 -9795> 2014-11-08 18:14:24.638178 7fc5b9328700 5 -- op tracker -- seq: 46675, time: 2014-11-08 18:14:24.638106, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) -9794> 2014-11-08 18:14:24.638183 7fc5b9328700 5 -- op tracker -- seq: 46675, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) -9793> 2014-11-08 18:14:24.638187 7fc5b9328700 5 -- op tracker -- seq: 46675, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) -9792> 2014-11-08 18:14:24.638190 7fc5b9328700 5 -- op tracker -- seq: 46675, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) -9791> 2014-11-08 18:14:24.638259 7fc5ad310700 5 -- op tracker -- seq: 46675, time: 2014-11-08 18:14:24.638259, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) -9790> 2014-11-08 18:14:24.666702 7fc5c362f700 5 osd.9 5773 tick -9789> 2014-11-08 18:14:24.736342 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) v1 -- ?+0 0x21a4bc00 con 0x42c86e0 -9788> 2014-11-08 18:14:24.736373 7fc5ad310700 5 -- op tracker -- seq: 46675, time: 2014-11-08 18:14:24.736373, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4383, to_read={7fd8e343/10000028cf9.00000005/head//4=0,4194304}, attrs_to_read=)) -9787> 2014-11-08 18:14:24.736378 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4bc00 con 0x42c86e0 -9786> 2014-11-08 18:14:24.736390 7fc5b9328700 5 -- op tracker -- seq: 46676, time: 2014-11-08 18:14:24.736366, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) -9785> 2014-11-08 18:14:24.736393 7fc5b9328700 5 -- op tracker -- seq: 46676, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) -9784> 2014-11-08 18:14:24.736396 7fc5b9328700 5 -- op tracker -- seq: 46676, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) -9783> 2014-11-08 18:14:24.736397 7fc5b9328700 5 -- op tracker -- seq: 46676, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) -9782> 2014-11-08 18:14:24.736421 7fc5afb15700 5 -- op tracker -- seq: 46676, time: 2014-11-08 18:14:24.736421, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) -9781> 2014-11-08 18:14:24.736453 7fc5afb15700 5 -- op tracker -- seq: 46676, time: 2014-11-08 18:14:24.736453, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=0)) -9780> 2014-11-08 18:14:24.760445 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8424 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) v3 ==== 757+0+0 (1959049713 0 0) 0x1e800900 con 0x1c5c0580 -9779> 2014-11-08 18:14:24.760486 7fc5a152e700 5 -- op tracker -- seq: 46677, time: 2014-11-08 18:14:24.760353, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9778> 2014-11-08 18:14:24.760501 7fc5a152e700 5 -- op tracker -- seq: 46677, time: 2014-11-08 18:14:24.760356, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9777> 2014-11-08 18:14:24.760510 7fc5a152e700 5 -- op tracker -- seq: 46677, time: 2014-11-08 18:14:24.760435, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9776> 2014-11-08 18:14:24.760519 7fc5a152e700 5 -- op tracker -- seq: 46677, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9775> 2014-11-08 18:14:24.760561 7fc5adb11700 5 -- op tracker -- seq: 46677, time: 2014-11-08 18:14:24.760561, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9774> 2014-11-08 18:14:24.760571 7fc5adb11700 5 -- op tracker -- seq: 46677, time: 2014-11-08 18:14:24.760571, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9773> 2014-11-08 18:14:24.760637 7fc5a152e700 5 -- op tracker -- seq: 46677, time: 2014-11-08 18:14:24.760637, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 6b50fcb5/1000001a6ca.0000003c/head//4) -9772> 2014-11-08 18:14:24.793092 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25f97da0 con 0x1c5c0580 -9771> 2014-11-08 18:14:24.793113 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25f90000 con 0x1c5c0580 -9770> 2014-11-08 18:14:24.793119 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1eae0d20 con 0x1c5c0580 -9769> 2014-11-08 18:14:24.793126 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1eae01e0 con 0x1c5c0580 -9768> 2014-11-08 18:14:24.793131 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1eae03c0 con 0x1c5c0580 -9767> 2014-11-08 18:14:24.801445 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=f68b1d83/1000001b477.000007ee/head//4,a21c1d83/1000001a103.00000095/head//4,42c1d83/1000001ab1a.00000015/head//4,b73c1d83/1000002a908.0000000a/head//4)) v1 -- ?+0 0x2562d340 con 0x1c5c0580 -9766> 2014-11-08 18:14:24.801469 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2562ee40 con 0x42c86e0 -9765> 2014-11-08 18:14:24.801485 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=4 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 885c1d83/1000002b519.00000021/head//4 -9764> 2014-11-08 18:14:24.801510 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb fa0b1d83/1000002a73b.00000147/head//4) v3 -- ?+0 0x1d3c8000 con 0x1e50aec0 -9763> 2014-11-08 18:14:24.801536 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2562ee40 con 0x42c86e0 -9762> 2014-11-08 18:14:24.801566 7fc5b9328700 5 -- op tracker -- seq: 46678, time: 2014-11-08 18:14:24.801479, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) -9761> 2014-11-08 18:14:24.801575 7fc5b9328700 5 -- op tracker -- seq: 46678, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) -9760> 2014-11-08 18:14:24.801581 7fc5b9328700 5 -- op tracker -- seq: 46678, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) -9759> 2014-11-08 18:14:24.801587 7fc5b9328700 5 -- op tracker -- seq: 46678, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) -9758> 2014-11-08 18:14:24.801619 7fc5b0316700 5 -- op tracker -- seq: 46678, time: 2014-11-08 18:14:24.801619, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) -9757> 2014-11-08 18:14:24.881617 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1560 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (1877075304 0 0) 0x22879f80 con 0x21468c60 -9756> 2014-11-08 18:14:24.881655 7fc5a39d2700 5 -- op tracker -- seq: 46679, time: 2014-11-08 18:14:24.881527, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) -9755> 2014-11-08 18:14:24.881667 7fc5a39d2700 5 -- op tracker -- seq: 46679, time: 2014-11-08 18:14:24.881530, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) -9754> 2014-11-08 18:14:24.881671 7fc5a39d2700 5 -- op tracker -- seq: 46679, time: 2014-11-08 18:14:24.881611, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) -9753> 2014-11-08 18:14:24.881676 7fc5a39d2700 5 -- op tracker -- seq: 46679, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) -9752> 2014-11-08 18:14:24.881724 7fc5af314700 5 -- op tracker -- seq: 46679, time: 2014-11-08 18:14:24.881724, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) -9751> 2014-11-08 18:14:24.971629 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59219, attrs_read=0)) v1 -- ?+0 0x224baa80 con 0x21468c60 -9750> 2014-11-08 18:14:24.971681 7fc5af314700 5 -- op tracker -- seq: 46679, time: 2014-11-08 18:14:24.971680, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59219, to_read={7b1d314e/1000002bb29.00000029/head//4=0,4194304}, attrs_to_read=)) -9749> 2014-11-08 18:14:25.123811 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) v1 -- ?+0 0x21839400 con 0x42c86e0 -9748> 2014-11-08 18:14:25.123869 7fc5b0316700 5 -- op tracker -- seq: 46678, time: 2014-11-08 18:14:25.123868, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4384, to_read={f68b1d83/1000001b477.000007ee/head//4=0,4194304,a21c1d83/1000001a103.00000095/head//4=0,4194304,42c1d83/1000001ab1a.00000015/head//4=0,4194304,b73c1d83/1000002a908.0000000a/head//4=0,4194304}, attrs_to_read=)) -9747> 2014-11-08 18:14:25.123881 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21839400 con 0x42c86e0 -9746> 2014-11-08 18:14:25.123894 7fc5b9328700 5 -- op tracker -- seq: 46680, time: 2014-11-08 18:14:25.123858, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) -9745> 2014-11-08 18:14:25.123898 7fc5b9328700 5 -- op tracker -- seq: 46680, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) -9744> 2014-11-08 18:14:25.123900 7fc5b9328700 5 -- op tracker -- seq: 46680, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) -9743> 2014-11-08 18:14:25.123902 7fc5b9328700 5 -- op tracker -- seq: 46680, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) -9742> 2014-11-08 18:14:25.123944 7fc5adb11700 5 -- op tracker -- seq: 46680, time: 2014-11-08 18:14:25.123944, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) -9741> 2014-11-08 18:14:25.123995 7fc5adb11700 5 -- op tracker -- seq: 46680, time: 2014-11-08 18:14:25.123995, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=0)) -9740> 2014-11-08 18:14:25.140330 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8425 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) v1 ==== 8390589+0+0 (765412865 0 0) 0x1c89a080 con 0x1c5c0580 -9739> 2014-11-08 18:14:25.140366 7fc5a152e700 5 -- op tracker -- seq: 46681, time: 2014-11-08 18:14:25.056576, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) -9738> 2014-11-08 18:14:25.140375 7fc5a152e700 5 -- op tracker -- seq: 46681, time: 2014-11-08 18:14:25.056578, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) -9737> 2014-11-08 18:14:25.140379 7fc5a152e700 5 -- op tracker -- seq: 46681, time: 2014-11-08 18:14:25.140318, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) -9736> 2014-11-08 18:14:25.140384 7fc5a152e700 5 -- op tracker -- seq: 46681, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) -9735> 2014-11-08 18:14:25.140421 7fc5b0316700 5 -- op tracker -- seq: 46681, time: 2014-11-08 18:14:25.140421, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) -9734> 2014-11-08 18:14:25.156072 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(f68b1d83/1000001b477.000007ee/head//4, version: 3033'2855, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f68b1d83/1000001b477.000007ee/head//4@3033'2855, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a21c1d83/1000001a103.00000095/head//4, version: 397'1769, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a21c1d83/1000001a103.00000095/head//4@397'1769, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c1d83/1000001ab1a.00000015/head//4, version: 2829'2462, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c1d83/1000001ab1a.00000015/head//4@2829'2462, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b73c1d83/1000002a908.0000000a/head//4, version: 3458'5383, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b73c1d83/1000002a908.0000000a/head/ /4@3458'5383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393ac00 con 0x1e50aec0 -9733> 2014-11-08 18:14:25.156173 7fc5b0316700 5 -- op tracker -- seq: 46681, time: 2014-11-08 18:14:25.156173, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4384, attrs_read=4)) -9732> 2014-11-08 18:14:25.595251 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3666 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:25.592994) v2 ==== 47+0+0 (4106033632 0 0) 0x239ee360 con 0x1cce3440 -9731> 2014-11-08 18:14:25.595285 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:25.592994) v2 -- ?+0 0x1db1d640 con 0x1cce3440 -9730> 2014-11-08 18:14:25.595308 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3666 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:25.592994) v2 ==== 47+0+0 (4106033632 0 0) 0x1d8ba760 con 0x1cce2ec0 -9729> 2014-11-08 18:14:25.595317 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:25.592994) v2 -- ?+0 0x2026eae0 con 0x1cce2ec0 -9728> 2014-11-08 18:14:25.666837 7fc5c362f700 5 osd.9 5773 tick -9727> 2014-11-08 18:14:25.666878 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -9726> 2014-11-08 18:14:25.666889 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(1 pgs tid 1323 v 0) v1 -- ?+0 0x25a03c80 con 0x44ed6a0 -9725> 2014-11-08 18:14:26.039569 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -9724> 2014-11-08 18:14:26.039603 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x1dbfe180 con 0x1cae2940 -9723> 2014-11-08 18:14:26.039633 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x246cdfa0 con 0x1cae2680 -9722> 2014-11-08 18:14:26.039644 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x25fd1e00 con 0x1c5c02c0 -9721> 2014-11-08 18:14:26.039654 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x25fd0f00 con 0x1cae0160 -9720> 2014-11-08 18:14:26.039664 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x227b23a0 con 0x1cae0f20 -9719> 2014-11-08 18:14:26.039674 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x25fc5a00 con 0x1cae0c60 -9718> 2014-11-08 18:14:26.039692 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x22a223a0 con 0x1cae23c0 -9717> 2014-11-08 18:14:26.039703 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x22a27bc0 con 0x1cae56a0 -9716> 2014-11-08 18:14:26.039712 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x25fd7260 con 0x1cbf9ce0 -9715> 2014-11-08 18:14:26.039722 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x1f4c4b00 con 0x1d603020 -9714> 2014-11-08 18:14:26.039731 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x2515e900 con 0x1dbd9b80 -9713> 2014-11-08 18:14:26.039740 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x2515c740 con 0x1d235d80 -9712> 2014-11-08 18:14:26.039751 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x257c2580 con 0x1cae1fa0 -9711> 2014-11-08 18:14:26.039760 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x257c4920 con 0x1cae1ce0 -9710> 2014-11-08 18:14:26.039769 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x21211c20 con 0x1cae1340 -9709> 2014-11-08 18:14:26.039778 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x25fc5280 con 0x1cae1080 -9708> 2014-11-08 18:14:26.039786 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x25fc3c00 con 0x1cae1a20 -9707> 2014-11-08 18:14:26.039796 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:26.039599) v2 -- ?+0 0x237183c0 con 0x1cae2c00 -9706> 2014-11-08 18:14:26.040476 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x2515aee0 con 0x1cae23c0 -9705> 2014-11-08 18:14:26.040517 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x229d4380 con 0x1cae1fa0 -9704> 2014-11-08 18:14:26.040529 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1dbfc380 con 0x1cae2c00 -9703> 2014-11-08 18:14:26.040538 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1d4f1c20 con 0x1cae2940 -9702> 2014-11-08 18:14:26.040548 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1f4c1680 con 0x1cae2680 -9701> 2014-11-08 18:14:26.040556 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x224579e0 con 0x1cae1340 -9700> 2014-11-08 18:14:26.040566 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1dece900 con 0x1cae1080 -9699> 2014-11-08 18:14:26.040574 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x223483c0 con 0x1cae1a20 -9698> 2014-11-08 18:14:26.040620 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1e8905a0 con 0x1cae0c60 -9697> 2014-11-08 18:14:26.040635 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x23d72940 con 0x1cae1ce0 -9696> 2014-11-08 18:14:26.040644 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 494 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1d6323a0 con 0x1cbf9ce0 -9695> 2014-11-08 18:14:26.040653 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x22e9f800 con 0x1cae0f20 -9694> 2014-11-08 18:14:26.040799 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 494 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x1dbfd460 con 0x1d603020 -9693> 2014-11-08 18:14:26.040820 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x227b6360 con 0x1c5c02c0 -9692> 2014-11-08 18:14:26.040856 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 301 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x24882580 con 0x1d235d80 -9691> 2014-11-08 18:14:26.040889 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x246cf620 con 0x1cae0160 -9690> 2014-11-08 18:14:26.040944 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 301 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x24880000 con 0x1dbd9b80 -9689> 2014-11-08 18:14:26.040958 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3714 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.039599) v2 ==== 47+0+0 (2250928920 0 0) 0x2583a760 con 0x1cae56a0 -9688> 2014-11-08 18:14:26.096104 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3667 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.093831) v2 ==== 47+0+0 (1726739862 0 0) 0x1d8b9e00 con 0x1cce2ec0 -9687> 2014-11-08 18:14:26.096128 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.093831) v2 -- ?+0 0x1d8ba760 con 0x1cce2ec0 -9686> 2014-11-08 18:14:26.096150 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3667 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.093831) v2 ==== 47+0+0 (1726739862 0 0) 0x239eeea0 con 0x1cce3440 -9685> 2014-11-08 18:14:26.096160 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.093831) v2 -- ?+0 0x239ee360 con 0x1cce3440 -9684> 2014-11-08 18:14:26.219152 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3728 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.209749) v2 ==== 47+0+0 (1521123436 0 0) 0x2515e360 con 0x1cce3700 -9683> 2014-11-08 18:14:26.219180 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.209749) v2 -- ?+0 0x239eeea0 con 0x1cce3700 -9682> 2014-11-08 18:14:26.219982 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3728 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.209749) v2 ==== 47+0+0 (1521123436 0 0) 0x25e84b00 con 0x1cce3180 -9681> 2014-11-08 18:14:26.220004 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.209749) v2 -- ?+0 0x1d8b9e00 con 0x1cce3180 -9680> 2014-11-08 18:14:26.419072 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1498 ==== pg_stats_ack(1 pgs tid 1323) v1 ==== 33+0+0 (3007575215 0 0) 0x23144b00 con 0x44ed6a0 -9679> 2014-11-08 18:14:26.421552 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(b6c3bddd/10000021770.00000000/head//4),PushReplyOp(e704bddd/10000016493.00000000/head//4),PushReplyOp(c714bddd/1000000ca62.00000000/head//4),PushReplyOp(df64bddd/100000296f3.00000095/head//4)]) v2 -- ?+0 0x1d5b1000 con 0x21468c60 -9678> 2014-11-08 18:14:26.445436 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(d5255b16/1000002bc82.0000012f/head//4),PushReplyOp(7cd55b16/10000019b02.0000032d/head//4),PushReplyOp(6f165b16/1000002911d.00000002/head//4),PushReplyOp(5c865b16/1000002a334.00000063/head//4),PushReplyOp(f9b65b16/10000019f97.00000010/head//4)]) v2 -- ?+0 0x229b8a00 con 0x1c5c0580 -9677> 2014-11-08 18:14:26.447629 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8426 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) v3 ==== 757+0+0 (1153762309 0 0) 0x1e800900 con 0x1c5c0580 -9676> 2014-11-08 18:14:26.447654 7fc5a152e700 5 -- op tracker -- seq: 46682, time: 2014-11-08 18:14:26.447575, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9675> 2014-11-08 18:14:26.447668 7fc5a152e700 5 -- op tracker -- seq: 46682, time: 2014-11-08 18:14:26.447578, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9674> 2014-11-08 18:14:26.447678 7fc5a152e700 5 -- op tracker -- seq: 46682, time: 2014-11-08 18:14:26.447624, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9673> 2014-11-08 18:14:26.447695 7fc5a152e700 5 -- op tracker -- seq: 46682, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9672> 2014-11-08 18:14:26.447754 7fc5ad310700 5 -- op tracker -- seq: 46682, time: 2014-11-08 18:14:26.447754, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9671> 2014-11-08 18:14:26.447766 7fc5ad310700 5 -- op tracker -- seq: 46682, time: 2014-11-08 18:14:26.447766, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9670> 2014-11-08 18:14:26.447821 7fc5ad310700 5 -- op tracker -- seq: 46682, time: 2014-11-08 18:14:26.447820, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f9b65b16/10000019f97.00000010/head//4) -9669> 2014-11-08 18:14:26.467652 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1561 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) v3 ==== 757+0+0 (1934264497 0 0) 0x1d653180 con 0x21468c60 -9668> 2014-11-08 18:14:26.467674 7fc5a39d2700 5 -- op tracker -- seq: 46683, time: 2014-11-08 18:14:26.467588, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9667> 2014-11-08 18:14:26.467679 7fc5a39d2700 5 -- op tracker -- seq: 46683, time: 2014-11-08 18:14:26.467590, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9666> 2014-11-08 18:14:26.467682 7fc5a39d2700 5 -- op tracker -- seq: 46683, time: 2014-11-08 18:14:26.467646, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9665> 2014-11-08 18:14:26.467692 7fc5a39d2700 5 -- op tracker -- seq: 46683, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9664> 2014-11-08 18:14:26.467734 7fc5af314700 5 -- op tracker -- seq: 46683, time: 2014-11-08 18:14:26.467734, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9663> 2014-11-08 18:14:26.467743 7fc5af314700 5 -- op tracker -- seq: 46683, time: 2014-11-08 18:14:26.467743, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9662> 2014-11-08 18:14:26.467789 7fc5af314700 5 -- op tracker -- seq: 46683, time: 2014-11-08 18:14:26.467789, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb df64bddd/100000296f3.00000095/head//4) -9661> 2014-11-08 18:14:26.489884 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 311 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.487569) v2 ==== 47+0+0 (103787515 0 0) 0x22e5b2a0 con 0x1eb9d6a0 -9660> 2014-11-08 18:14:26.489914 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.487569) v2 -- ?+0 0x25e84b00 con 0x1eb9d6a0 -9659> 2014-11-08 18:14:26.489957 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 311 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.487569) v2 ==== 47+0+0 (103787515 0 0) 0x2026a1c0 con 0x1eb9c780 -9658> 2014-11-08 18:14:26.489969 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.487569) v2 -- ?+0 0x2515e360 con 0x1eb9c780 -9657> 2014-11-08 18:14:26.666986 7fc5c362f700 5 osd.9 5773 tick -9656> 2014-11-08 18:14:26.691966 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 489 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.690143) v2 ==== 47+0+0 (1175911086 0 0) 0x22d85640 con 0x1eb9c8e0 -9655> 2014-11-08 18:14:26.692005 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.690143) v2 -- ?+0 0x2026a1c0 con 0x1eb9c8e0 -9654> 2014-11-08 18:14:26.692635 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 489 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:26.690143) v2 ==== 47+0+0 (1175911086 0 0) 0x1d076360 con 0x1eb9d800 -9653> 2014-11-08 18:14:26.692656 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:26.690143) v2 -- ?+0 0x22e5b2a0 con 0x1eb9d800 -9652> 2014-11-08 18:14:27.086776 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8427 ==== MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1068296594 0 0) 0x229b8a00 con 0x1c5c0580 -9651> 2014-11-08 18:14:27.086818 7fc5a152e700 5 -- op tracker -- seq: 46684, time: 2014-11-08 18:14:27.003586, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9650> 2014-11-08 18:14:27.086832 7fc5a152e700 5 -- op tracker -- seq: 46684, time: 2014-11-08 18:14:27.003591, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9649> 2014-11-08 18:14:27.086840 7fc5a152e700 5 -- op tracker -- seq: 46684, time: 2014-11-08 18:14:27.086766, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9648> 2014-11-08 18:14:27.086848 7fc5a152e700 5 -- op tracker -- seq: 46684, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9647> 2014-11-08 18:14:27.086914 7fc5ad310700 5 -- op tracker -- seq: 46684, time: 2014-11-08 18:14:27.086914, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9646> 2014-11-08 18:14:27.086996 7fc5ad310700 5 -- op tracker -- seq: 46684, time: 2014-11-08 18:14:27.086995, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(9a375b16/1000001b38e.00000033/head//4, version: 3033'2797, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9a375b16/1000001b38e.00000033/head//4@3033'2797, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9645> 2014-11-08 18:14:27.095461 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2144 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) v1 ==== 2097696+0+0 (902895738 0 0) 0x21a4d780 con 0x1e50aec0 -9644> 2014-11-08 18:14:27.095499 7fc5a56ef700 5 -- op tracker -- seq: 46685, time: 2014-11-08 18:14:25.996754, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) -9643> 2014-11-08 18:14:27.095513 7fc5a56ef700 5 -- op tracker -- seq: 46685, time: 2014-11-08 18:14:25.996783, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) -9642> 2014-11-08 18:14:27.095522 7fc5a56ef700 5 -- op tracker -- seq: 46685, time: 2014-11-08 18:14:27.095452, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) -9641> 2014-11-08 18:14:27.095531 7fc5a56ef700 5 -- op tracker -- seq: 46685, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) -9640> 2014-11-08 18:14:27.095574 7fc5afb15700 5 -- op tracker -- seq: 46685, time: 2014-11-08 18:14:27.095574, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) -9639> 2014-11-08 18:14:27.097748 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1562 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2708613287 0 0) 0x22879d40 con 0x21468c60 -9638> 2014-11-08 18:14:27.097770 7fc5a39d2700 5 -- op tracker -- seq: 46686, time: 2014-11-08 18:14:27.097680, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) -9637> 2014-11-08 18:14:27.097777 7fc5a39d2700 5 -- op tracker -- seq: 46686, time: 2014-11-08 18:14:27.097690, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) -9636> 2014-11-08 18:14:27.097780 7fc5a39d2700 5 -- op tracker -- seq: 46686, time: 2014-11-08 18:14:27.097742, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) -9635> 2014-11-08 18:14:27.097783 7fc5a39d2700 5 -- op tracker -- seq: 46686, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) -9634> 2014-11-08 18:14:27.098906 7fc5acb0f700 5 -- op tracker -- seq: 46686, time: 2014-11-08 18:14:27.098906, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) -9633> 2014-11-08 18:14:27.105479 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(7fd8e343/10000028cf9.00000005/head//4, version: 3033'7763, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fd8e343/10000028cf9.00000005/head//4@3033'7763, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229bb200 con 0x1c5c0580 -9632> 2014-11-08 18:14:27.108735 7fc5afb15700 5 -- op tracker -- seq: 46685, time: 2014-11-08 18:14:27.108734, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4383, attrs_read=1)) -9631> 2014-11-08 18:14:27.137176 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1eae1860 con 0x1c5c0580 -9630> 2014-11-08 18:14:27.137224 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(9a375b16/1000001b38e.00000033/head//4)]) v2 -- ?+0 0x1d3aba00 con 0x1c5c0580 -9629> 2014-11-08 18:14:27.170794 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8428 ==== MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3031416852 0 0) 0x229b8600 con 0x1c5c0580 -9628> 2014-11-08 18:14:27.170835 7fc5a152e700 5 -- op tracker -- seq: 46687, time: 2014-11-08 18:14:27.086909, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9627> 2014-11-08 18:14:27.170850 7fc5a152e700 5 -- op tracker -- seq: 46687, time: 2014-11-08 18:14:27.086910, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9626> 2014-11-08 18:14:27.170859 7fc5a152e700 5 -- op tracker -- seq: 46687, time: 2014-11-08 18:14:27.170786, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9625> 2014-11-08 18:14:27.170867 7fc5a152e700 5 -- op tracker -- seq: 46687, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9624> 2014-11-08 18:14:27.170912 7fc5adb11700 5 -- op tracker -- seq: 46687, time: 2014-11-08 18:14:27.170912, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9623> 2014-11-08 18:14:27.170993 7fc5adb11700 5 -- op tracker -- seq: 46687, time: 2014-11-08 18:14:27.170993, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(c3e0fcb5/10000029ef9.00000059/head//4, version: 3033'4612, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c3e0fcb5/10000029ef9.00000059/head//4@3033'4612, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9622> 2014-11-08 18:14:27.171948 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8429 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) v3 ==== 757+0+0 (2417723400 0 0) 0x248b1f80 con 0x1c5c0580 -9621> 2014-11-08 18:14:27.171969 7fc5a152e700 5 -- op tracker -- seq: 46688, time: 2014-11-08 18:14:27.171886, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9620> 2014-11-08 18:14:27.171975 7fc5a152e700 5 -- op tracker -- seq: 46688, time: 2014-11-08 18:14:27.171889, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9619> 2014-11-08 18:14:27.171978 7fc5a152e700 5 -- op tracker -- seq: 46688, time: 2014-11-08 18:14:27.171943, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9618> 2014-11-08 18:14:27.171981 7fc5a152e700 5 -- op tracker -- seq: 46688, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9617> 2014-11-08 18:14:27.172015 7fc5ad310700 5 -- op tracker -- seq: 46688, time: 2014-11-08 18:14:27.172015, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9616> 2014-11-08 18:14:27.172023 7fc5ad310700 5 -- op tracker -- seq: 46688, time: 2014-11-08 18:14:27.172023, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9615> 2014-11-08 18:14:27.172068 7fc5a152e700 5 -- op tracker -- seq: 46688, time: 2014-11-08 18:14:27.172068, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 9a375b16/1000001b38e.00000033/head//4) -9614> 2014-11-08 18:14:27.172121 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8430 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x1eae1860 con 0x1c5c0580 -9613> 2014-11-08 18:14:27.172135 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172081, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9612> 2014-11-08 18:14:27.172139 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172082, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9611> 2014-11-08 18:14:27.172142 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172105, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9610> 2014-11-08 18:14:27.172145 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172131, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9609> 2014-11-08 18:14:27.172148 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172148, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9608> 2014-11-08 18:14:27.172154 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172154, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9607> 2014-11-08 18:14:27.172158 7fc5b9b29700 5 -- op tracker -- seq: 46689, time: 2014-11-08 18:14:27.172158, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9606> 2014-11-08 18:14:27.270140 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x4536900 con 0x1c5c0580 -9605> 2014-11-08 18:14:27.270210 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(c3e0fcb5/10000029ef9.00000059/head//4)]) v2 -- ?+0 0x23939000 con 0x1c5c0580 -9604> 2014-11-08 18:14:27.273515 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8431 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) v3 ==== 757+0+0 (3585937512 0 0) 0x248b1f80 con 0x1c5c0580 -9603> 2014-11-08 18:14:27.273537 7fc5a152e700 5 -- op tracker -- seq: 46690, time: 2014-11-08 18:14:27.273468, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9602> 2014-11-08 18:14:27.273564 7fc5a152e700 5 -- op tracker -- seq: 46690, time: 2014-11-08 18:14:27.273470, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9601> 2014-11-08 18:14:27.273570 7fc5a152e700 5 -- op tracker -- seq: 46690, time: 2014-11-08 18:14:27.273510, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9600> 2014-11-08 18:14:27.273573 7fc5a152e700 5 -- op tracker -- seq: 46690, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9599> 2014-11-08 18:14:27.273617 7fc5b0316700 5 -- op tracker -- seq: 46690, time: 2014-11-08 18:14:27.273617, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9598> 2014-11-08 18:14:27.273628 7fc5b0316700 5 -- op tracker -- seq: 46690, time: 2014-11-08 18:14:27.273628, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9597> 2014-11-08 18:14:27.273678 7fc5b0316700 5 -- op tracker -- seq: 46690, time: 2014-11-08 18:14:27.273678, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb c3e0fcb5/10000029ef9.00000059/head//4) -9596> 2014-11-08 18:14:27.417752 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1563 ==== MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4e45bddd/ 1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8392121+0+0 (75213876 0 0) 0x1d5b1000 con 0x21468c60 -9595> 2014-11-08 18:14:27.417807 7fc5a39d2700 5 -- op tracker -- seq: 46691, time: 2014-11-08 18:14:27.157508, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 4e45bddd/1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9594> 2014-11-08 18:14:27.417828 7fc5a39d2700 5 -- op tracker -- seq: 46691, time: 2014-11-08 18:14:27.157512, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 4e45bddd/1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9593> 2014-11-08 18:14:27.417845 7fc5a39d2700 5 -- op tracker -- seq: 46691, time: 2014-11-08 18:14:27.417719, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 4e45bddd/1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9592> 2014-11-08 18:14:27.417861 7fc5a39d2700 5 -- op tracker -- seq: 46691, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4e45bddd/ 1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9591> 2014-11-08 18:14:27.417910 7fc5af314700 5 -- op tracker -- seq: 46691, time: 2014-11-08 18:14:27.417910, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 4e45bddd/1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9590> 2014-11-08 18:14:27.418094 7fc5af314700 5 -- op tracker -- seq: 46691, time: 2014-11-08 18:14:27.418094, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(e294bddd/1000002a327.0000000c/head//4, version: 3033'9334, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e294bddd/1000002a327.0000000c/head//4@3033'9334, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6dc4bddd/1000002ae95.00000890/head//4, version: 3468'12200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dc4bddd/1000002ae95.00000890/head//4@3468'12200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(79f4bddd/10000019b35.0000009f/head//4, version: 397'2148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(79f4bddd/10000019b35.0000009f/head//4@397'2148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4e45bddd/1000002861c.00000021/head//4, version: 3033'7468, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4e45bddd/ 1000002861c.00000021/head//4@3033'7468, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9589> 2014-11-08 18:14:27.420073 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2145 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x1decfda0 con 0x1e50aec0 -9588> 2014-11-08 18:14:27.420111 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.419882, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -9587> 2014-11-08 18:14:27.420119 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.419885, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -9586> 2014-11-08 18:14:27.420122 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.419953, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -9585> 2014-11-08 18:14:27.420125 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.420105, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -9584> 2014-11-08 18:14:27.420128 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.420128, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -9583> 2014-11-08 18:14:27.420138 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.420138, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -9582> 2014-11-08 18:14:27.420144 7fc5b9b29700 5 -- op tracker -- seq: 46692, time: 2014-11-08 18:14:27.420144, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -9581> 2014-11-08 18:14:27.420157 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2146 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x1caaf620 con 0x1e50aec0 -9580> 2014-11-08 18:14:27.420164 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.419971, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -9579> 2014-11-08 18:14:27.420167 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.419972, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -9578> 2014-11-08 18:14:27.420170 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.420000, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -9577> 2014-11-08 18:14:27.420172 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.420163, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -9576> 2014-11-08 18:14:27.420175 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.420175, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -9575> 2014-11-08 18:14:27.420179 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.420179, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -9574> 2014-11-08 18:14:27.420182 7fc5b9b29700 5 -- op tracker -- seq: 46693, time: 2014-11-08 18:14:27.420182, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -9573> 2014-11-08 18:14:27.420188 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2147 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x1dec8b40 con 0x1e50aec0 -9572> 2014-11-08 18:14:27.420194 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420005, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -9571> 2014-11-08 18:14:27.420197 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420005, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -9570> 2014-11-08 18:14:27.420200 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420022, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -9569> 2014-11-08 18:14:27.420203 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420194, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -9568> 2014-11-08 18:14:27.420206 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420205, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -9567> 2014-11-08 18:14:27.420209 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420209, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -9566> 2014-11-08 18:14:27.420212 7fc5b9b29700 5 -- op tracker -- seq: 46694, time: 2014-11-08 18:14:27.420212, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -9565> 2014-11-08 18:14:27.420218 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2148 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x1decbde0 con 0x1e50aec0 -9564> 2014-11-08 18:14:27.420224 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420027, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -9563> 2014-11-08 18:14:27.420227 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420027, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -9562> 2014-11-08 18:14:27.420230 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420045, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -9561> 2014-11-08 18:14:27.420233 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420224, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -9560> 2014-11-08 18:14:27.420236 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420236, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -9559> 2014-11-08 18:14:27.420239 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420239, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -9558> 2014-11-08 18:14:27.420243 7fc5b9b29700 5 -- op tracker -- seq: 46695, time: 2014-11-08 18:14:27.420243, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -9557> 2014-11-08 18:14:27.543441 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8432 ==== MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2293090427 0 0) 0x23939000 con 0x1c5c0580 -9556> 2014-11-08 18:14:27.545710 7fc5a152e700 5 -- op tracker -- seq: 46696, time: 2014-11-08 18:14:27.474875, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9555> 2014-11-08 18:14:27.546559 7fc5a152e700 5 -- op tracker -- seq: 46696, time: 2014-11-08 18:14:27.474881, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9554> 2014-11-08 18:14:27.546576 7fc5a152e700 5 -- op tracker -- seq: 46696, time: 2014-11-08 18:14:27.543431, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9553> 2014-11-08 18:14:27.546590 7fc5a152e700 5 -- op tracker -- seq: 46696, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9552> 2014-11-08 18:14:27.546736 7fc5adb11700 5 -- op tracker -- seq: 46696, time: 2014-11-08 18:14:27.546736, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9551> 2014-11-08 18:14:27.546821 7fc5adb11700 5 -- op tracker -- seq: 46696, time: 2014-11-08 18:14:27.546821, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(acc1fcb5/10000019a3f.00000067/head//4, version: 397'1076, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(acc1fcb5/10000019a3f.00000067/head//4@397'1076, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9550> 2014-11-08 18:14:27.667336 7fc5c362f700 5 osd.9 5773 tick -9549> 2014-11-08 18:14:27.910736 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3729 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:27.910373) v2 ==== 47+0+0 (2921043038 0 0) 0x251585a0 con 0x1cce3700 -9548> 2014-11-08 18:14:27.910767 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:27.910373) v2 -- ?+0 0x22d85640 con 0x1cce3700 -9547> 2014-11-08 18:14:27.910861 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3729 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:27.910373) v2 ==== 47+0+0 (2921043038 0 0) 0x25e85fa0 con 0x1cce3180 -9546> 2014-11-08 18:14:27.910876 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:27.910373) v2 -- ?+0 0x1d076360 con 0x1cce3180 -9545> 2014-11-08 18:14:28.205715 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x4535280 con 0x21468c60 -9544> 2014-11-08 18:14:28.205760 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x45350a0 con 0x21468c60 -9543> 2014-11-08 18:14:28.205767 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x23c30b40 con 0x21468c60 -9542> 2014-11-08 18:14:28.205773 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x23c31e00 con 0x21468c60 -9541> 2014-11-08 18:14:28.205780 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(e294bddd/1000002a327.0000000c/head//4),PushReplyOp(6dc4bddd/1000002ae95.00000890/head//4),PushReplyOp(79f4bddd/10000019b35.0000009f/head//4),PushReplyOp(4e45bddd/1000002861c.00000021/head//4)]) v2 -- ?+0 0x1d5b3c00 con 0x21468c60 -9540> 2014-11-08 18:14:28.254717 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x23c36360 con 0x1c5c0580 -9539> 2014-11-08 18:14:28.254761 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(acc1fcb5/10000019a3f.00000067/head//4)]) v2 -- ?+0 0x229b8600 con 0x1c5c0580 -9538> 2014-11-08 18:14:28.307637 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1564 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) v3 ==== 757+0+0 (4062999162 0 0) 0x1d653f00 con 0x21468c60 -9537> 2014-11-08 18:14:28.307666 7fc5a39d2700 5 -- op tracker -- seq: 46697, time: 2014-11-08 18:14:28.307562, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9536> 2014-11-08 18:14:28.307678 7fc5a39d2700 5 -- op tracker -- seq: 46697, time: 2014-11-08 18:14:28.307564, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9535> 2014-11-08 18:14:28.307693 7fc5a39d2700 5 -- op tracker -- seq: 46697, time: 2014-11-08 18:14:28.307617, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9534> 2014-11-08 18:14:28.307704 7fc5a39d2700 5 -- op tracker -- seq: 46697, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9533> 2014-11-08 18:14:28.307769 7fc5af314700 5 -- op tracker -- seq: 46697, time: 2014-11-08 18:14:28.307769, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9532> 2014-11-08 18:14:28.307780 7fc5af314700 5 -- op tracker -- seq: 46697, time: 2014-11-08 18:14:28.307779, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9531> 2014-11-08 18:14:28.307820 7fc5af314700 5 -- op tracker -- seq: 46697, time: 2014-11-08 18:14:28.307820, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 4e45bddd/1000002861c.00000021/head//4) -9530> 2014-11-08 18:14:28.432141 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8433 ==== MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/1000002858f. 0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8395472+0+0 (2808098822 0 0) 0x1d3aba00 con 0x1c5c0580 -9529> 2014-11-08 18:14:28.432206 7fc5a152e700 5 -- op tracker -- seq: 46698, time: 2014-11-08 18:14:28.254019, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/ 1000002858f.0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9528> 2014-11-08 18:14:28.432237 7fc5a152e700 5 -- op tracker -- seq: 46698, time: 2014-11-08 18:14:28.254022, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/ 1000002858f.0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9527> 2014-11-08 18:14:28.432262 7fc5a152e700 5 -- op tracker -- seq: 46698, time: 2014-11-08 18:14:28.432129, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/ 1000002858f.0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9526> 2014-11-08 18:14:28.432286 7fc5a152e700 5 -- op tracker -- seq: 46698, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/1000002858f. 0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9525> 2014-11-08 18:14:28.432385 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8434 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) v3 ==== 757+0+0 (2850612040 0 0) 0x1d33ad00 con 0x1c5c0580 -9524> 2014-11-08 18:14:28.432401 7fc5a152e700 5 -- op tracker -- seq: 46699, time: 2014-11-08 18:14:28.432352, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9523> 2014-11-08 18:14:28.432411 7fc5a152e700 5 -- op tracker -- seq: 46699, time: 2014-11-08 18:14:28.432353, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9522> 2014-11-08 18:14:28.432420 7fc5a152e700 5 -- op tracker -- seq: 46699, time: 2014-11-08 18:14:28.432384, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9521> 2014-11-08 18:14:28.432429 7fc5a152e700 5 -- op tracker -- seq: 46699, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9520> 2014-11-08 18:14:28.432502 7fc5afb15700 5 -- op tracker -- seq: 46698, time: 2014-11-08 18:14:28.432502, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/ 1000002858f.0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9519> 2014-11-08 18:14:28.432680 7fc5afb15700 5 -- op tracker -- seq: 46698, time: 2014-11-08 18:14:28.432680, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(89575b16/1000002bbf8.00000011/head//4, version: 3478'7567, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(89575b16/1000002bbf8.00000011/head//4@3478'7567, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d585b16/1000002bdb5.00000003/head//4, version: 4864'8336, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d585b16/1000002bdb5.00000003/head//4@4864'8336, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d4d85b16/1000000f123.00000000/head//4, version: 397'380, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d4d85b16/1000000f123.00000000/head//4@397'380, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45d95b16/1000002858f.0000000d/head//4, version: 3033'3809, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45d95b16/1000002858f. 0000000d/head//4@3033'3809, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a5e95b16/1000002b3ba.00000045/head//4, version: 3468'6394, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a5e95b16/1000002b3ba.00000045/head//4@3468'6394, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9518> 2014-11-08 18:14:28.432730 7fc5b0316700 5 -- op tracker -- seq: 46699, time: 2014-11-08 18:14:28.432730, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9517> 2014-11-08 18:14:28.432738 7fc5b0316700 5 -- op tracker -- seq: 46699, time: 2014-11-08 18:14:28.432737, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9516> 2014-11-08 18:14:28.432765 7fc5b0316700 5 -- op tracker -- seq: 46699, time: 2014-11-08 18:14:28.432764, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb acc1fcb5/10000019a3f.00000067/head//4) -9515> 2014-11-08 18:14:28.561760 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8435 ==== MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2637849086 0 0) 0x229b8600 con 0x1c5c0580 -9514> 2014-11-08 18:14:28.561808 7fc5a152e700 5 -- op tracker -- seq: 46700, time: 2014-11-08 18:14:28.494081, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9513> 2014-11-08 18:14:28.561827 7fc5a152e700 5 -- op tracker -- seq: 46700, time: 2014-11-08 18:14:28.494086, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9512> 2014-11-08 18:14:28.561839 7fc5a152e700 5 -- op tracker -- seq: 46700, time: 2014-11-08 18:14:28.561732, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9511> 2014-11-08 18:14:28.561851 7fc5a152e700 5 -- op tracker -- seq: 46700, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9510> 2014-11-08 18:14:28.561895 7fc5adb11700 5 -- op tracker -- seq: 46700, time: 2014-11-08 18:14:28.561895, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9509> 2014-11-08 18:14:28.563647 7fc5a152e700 5 -- op tracker -- seq: 46700, time: 2014-11-08 18:14:28.563647, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(16d1fcb5/1000001b476.0000002b/head//4, version: 3033'2976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16d1fcb5/1000001b476.0000002b/head//4@3033'2976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9508> 2014-11-08 18:14:28.603653 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x23c341a0 con 0x1c5c0580 -9507> 2014-11-08 18:14:28.603702 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x23c305a0 con 0x1c5c0580 -9506> 2014-11-08 18:14:28.603709 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x23c37620 con 0x1c5c0580 -9505> 2014-11-08 18:14:28.603715 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2286ddc0 con 0x1c5c0580 -9504> 2014-11-08 18:14:28.603721 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2286e540 con 0x1c5c0580 -9503> 2014-11-08 18:14:28.603729 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(89575b16/1000002bbf8.00000011/head//4),PushReplyOp(8d585b16/1000002bdb5.00000003/head//4),PushReplyOp(d4d85b16/1000000f123.00000000/head//4),PushReplyOp(45d95b16/1000002858f.0000000d/head//4),PushReplyOp(a5e95b16/1000002b3ba.00000045/head//4)]) v2 -- ?+0 0x229baa00 con 0x1c5c0580 -9502> 2014-11-08 18:14:28.609973 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8436 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) v3 ==== 757+0+0 (1324439277 0 0) 0x1d33f980 con 0x1c5c0580 -9501> 2014-11-08 18:14:28.610015 7fc5a152e700 5 -- op tracker -- seq: 46701, time: 2014-11-08 18:14:28.609862, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9500> 2014-11-08 18:14:28.610030 7fc5a152e700 5 -- op tracker -- seq: 46701, time: 2014-11-08 18:14:28.609865, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9499> 2014-11-08 18:14:28.610040 7fc5a152e700 5 -- op tracker -- seq: 46701, time: 2014-11-08 18:14:28.609945, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9498> 2014-11-08 18:14:28.610050 7fc5a152e700 5 -- op tracker -- seq: 46701, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9497> 2014-11-08 18:14:28.610094 7fc5ad310700 5 -- op tracker -- seq: 46701, time: 2014-11-08 18:14:28.610094, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9496> 2014-11-08 18:14:28.610103 7fc5ad310700 5 -- op tracker -- seq: 46701, time: 2014-11-08 18:14:28.610103, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9495> 2014-11-08 18:14:28.610328 7fc5a152e700 5 -- op tracker -- seq: 46701, time: 2014-11-08 18:14:28.610328, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb a5e95b16/1000002b3ba.00000045/head//4) -9494> 2014-11-08 18:14:28.653831 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2286d640 con 0x1c5c0580 -9493> 2014-11-08 18:14:28.653945 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(16d1fcb5/1000001b476.0000002b/head//4)]) v2 -- ?+0 0x23939000 con 0x1c5c0580 -9492> 2014-11-08 18:14:28.660358 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8437 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) v3 ==== 757+0+0 (1883797765 0 0) 0x1d33f980 con 0x1c5c0580 -9491> 2014-11-08 18:14:28.660391 7fc5a152e700 5 -- op tracker -- seq: 46702, time: 2014-11-08 18:14:28.660269, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9490> 2014-11-08 18:14:28.660406 7fc5a152e700 5 -- op tracker -- seq: 46702, time: 2014-11-08 18:14:28.660273, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9489> 2014-11-08 18:14:28.660416 7fc5a152e700 5 -- op tracker -- seq: 46702, time: 2014-11-08 18:14:28.660338, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9488> 2014-11-08 18:14:28.660444 7fc5a152e700 5 -- op tracker -- seq: 46702, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9487> 2014-11-08 18:14:28.660490 7fc5b0316700 5 -- op tracker -- seq: 46702, time: 2014-11-08 18:14:28.660490, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9486> 2014-11-08 18:14:28.660502 7fc5b0316700 5 -- op tracker -- seq: 46702, time: 2014-11-08 18:14:28.660502, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9485> 2014-11-08 18:14:28.660550 7fc5b0316700 5 -- op tracker -- seq: 46702, time: 2014-11-08 18:14:28.660550, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 16d1fcb5/1000001b476.0000002b/head//4) -9484> 2014-11-08 18:14:28.669906 7fc5c362f700 5 osd.9 5773 tick -9483> 2014-11-08 18:14:28.739545 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3699 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:28.737051) v2 ==== 47+0+0 (3347540419 0 0) 0x234e3c00 con 0x1c8b7d20 -9482> 2014-11-08 18:14:28.739574 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:28.737051) v2 -- ?+0 0x251585a0 con 0x1c8b7d20 -9481> 2014-11-08 18:14:28.739612 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3699 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:28.737051) v2 ==== 47+0+0 (3347540419 0 0) 0x2026e900 con 0x1c9f8580 -9480> 2014-11-08 18:14:28.739626 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:28.737051) v2 -- ?+0 0x25e85fa0 con 0x1c9f8580 -9479> 2014-11-08 18:14:28.766064 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8438 ==== MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 3229+0+0 (3205945101 0 0) 0x23939000 con 0x1c5c0580 -9478> 2014-11-08 18:14:28.766114 7fc5a152e700 5 -- op tracker -- seq: 46703, time: 2014-11-08 18:14:28.765903, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9477> 2014-11-08 18:14:28.766132 7fc5a152e700 5 -- op tracker -- seq: 46703, time: 2014-11-08 18:14:28.765908, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9476> 2014-11-08 18:14:28.766145 7fc5a152e700 5 -- op tracker -- seq: 46703, time: 2014-11-08 18:14:28.766040, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9475> 2014-11-08 18:14:28.766157 7fc5a152e700 5 -- op tracker -- seq: 46703, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9474> 2014-11-08 18:14:28.766205 7fc5afb15700 5 -- op tracker -- seq: 46703, time: 2014-11-08 18:14:28.766205, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9473> 2014-11-08 18:14:28.766791 7fc5a152e700 5 -- op tracker -- seq: 46703, time: 2014-11-08 18:14:28.766791, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(66e95b16/1000000965c.00000000/head//4, version: 397'195, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(66e95b16/1000000965c.00000000/head//4@397'195, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9472> 2014-11-08 18:14:28.795612 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2286fbc0 con 0x1c5c0580 -9471> 2014-11-08 18:14:28.795658 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(66e95b16/1000000965c.00000000/head//4)]) v2 -- ?+0 0x1d3aba00 con 0x1c5c0580 -9470> 2014-11-08 18:14:28.797228 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8439 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) v3 ==== 757+0+0 (3265120808 0 0) 0x1d33e300 con 0x1c5c0580 -9469> 2014-11-08 18:14:28.797249 7fc5a152e700 5 -- op tracker -- seq: 46704, time: 2014-11-08 18:14:28.797175, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9468> 2014-11-08 18:14:28.797264 7fc5a152e700 5 -- op tracker -- seq: 46704, time: 2014-11-08 18:14:28.797177, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9467> 2014-11-08 18:14:28.797274 7fc5a152e700 5 -- op tracker -- seq: 46704, time: 2014-11-08 18:14:28.797215, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9466> 2014-11-08 18:14:28.797283 7fc5a152e700 5 -- op tracker -- seq: 46704, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9465> 2014-11-08 18:14:28.797320 7fc5ad310700 5 -- op tracker -- seq: 46704, time: 2014-11-08 18:14:28.797320, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9464> 2014-11-08 18:14:28.797330 7fc5ad310700 5 -- op tracker -- seq: 46704, time: 2014-11-08 18:14:28.797330, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9463> 2014-11-08 18:14:28.797551 7fc5a152e700 5 -- op tracker -- seq: 46704, time: 2014-11-08 18:14:28.797551, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 66e95b16/1000000965c.00000000/head//4) -9462> 2014-11-08 18:14:29.553357 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1565 ==== MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8392121+0+0 (2152642865 0 0) 0x1d5b3c00 con 0x21468c60 -9461> 2014-11-08 18:14:29.553414 7fc5a39d2700 5 -- op tracker -- seq: 46705, time: 2014-11-08 18:14:29.286478, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9460> 2014-11-08 18:14:29.553441 7fc5a39d2700 5 -- op tracker -- seq: 46705, time: 2014-11-08 18:14:29.286484, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9459> 2014-11-08 18:14:29.553462 7fc5a39d2700 5 -- op tracker -- seq: 46705, time: 2014-11-08 18:14:29.553322, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9458> 2014-11-08 18:14:29.553484 7fc5a39d2700 5 -- op tracker -- seq: 46705, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9457> 2014-11-08 18:14:29.553536 7fc5af314700 5 -- op tracker -- seq: 46705, time: 2014-11-08 18:14:29.553536, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9456> 2014-11-08 18:14:29.561150 7fc5a39d2700 5 -- op tracker -- seq: 46705, time: 2014-11-08 18:14:29.561149, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(bba5bddd/1000001a02d.00000008/head//4, version: 397'3189, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bba5bddd/1000001a02d.00000008/head//4@397'3189, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(656bddd/1000002a6bd.00000025/head//4, version: 3458'10171, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(656bddd/1000002a6bd.00000025/head//4@3458'10171, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86c6bddd/1000002b803.0000002b/head//4, version: 3473'13713, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86c6bddd/1000002b803.0000002b/head//4@3473'13713, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ca7bddd/10000019741.0000000b/head//4, version: 397'1431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6ca7bddd/10000019741.0000000b/head//4@397'1431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9455> 2014-11-08 18:14:29.670062 7fc5c362f700 5 osd.9 5773 tick -9454> 2014-11-08 18:14:29.840194 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3700 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:29.837932) v2 ==== 47+0+0 (2559816821 0 0) 0x21215a00 con 0x1c8b7d20 -9453> 2014-11-08 18:14:29.840239 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:29.837932) v2 -- ?+0 0x234e3c00 con 0x1c8b7d20 -9452> 2014-11-08 18:14:29.840478 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3700 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:29.837932) v2 ==== 47+0+0 (2559816821 0 0) 0x2026ad00 con 0x1c9f8580 -9451> 2014-11-08 18:14:29.840492 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:29.837932) v2 -- ?+0 0x2026e900 con 0x1c9f8580 -9450> 2014-11-08 18:14:29.921818 7fc5b2b1b700 10 monclient: tick -9449> 2014-11-08 18:14:29.921836 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:13:59.921835) -9448> 2014-11-08 18:14:29.921844 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:14:29.921844; renew after: 2014-11-08 18:16:29.290049) -- no -9447> 2014-11-08 18:14:30.021279 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x2286d820 con 0x21468c60 -9446> 2014-11-08 18:14:30.021311 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x2286dbe0 con 0x21468c60 -9445> 2014-11-08 18:14:30.021320 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x2515f260 con 0x21468c60 -9444> 2014-11-08 18:14:30.021328 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x258390e0 con 0x21468c60 -9443> 2014-11-08 18:14:30.021338 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(bba5bddd/1000001a02d.00000008/head//4),PushReplyOp(656bddd/1000002a6bd.00000025/head//4),PushReplyOp(86c6bddd/1000002b803.0000002b/head//4),PushReplyOp(6ca7bddd/10000019741.0000000b/head//4)]) v2 -- ?+0 0x1d5b1000 con 0x21468c60 -9442> 2014-11-08 18:14:30.072857 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8440 ==== MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3020246041 0 0) 0x1d3aba00 con 0x1c5c0580 -9441> 2014-11-08 18:14:30.072908 7fc5a152e700 5 -- op tracker -- seq: 46706, time: 2014-11-08 18:14:30.049385, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9440> 2014-11-08 18:14:30.072926 7fc5a152e700 5 -- op tracker -- seq: 46706, time: 2014-11-08 18:14:30.049388, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9439> 2014-11-08 18:14:30.072939 7fc5a152e700 5 -- op tracker -- seq: 46706, time: 2014-11-08 18:14:30.072831, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9438> 2014-11-08 18:14:30.072952 7fc5a152e700 5 -- op tracker -- seq: 46706, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9437> 2014-11-08 18:14:30.072999 7fc5afb15700 5 -- op tracker -- seq: 46706, time: 2014-11-08 18:14:30.072999, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9436> 2014-11-08 18:14:30.079272 7fc5a152e700 5 -- op tracker -- seq: 46706, time: 2014-11-08 18:14:30.079272, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(161a5b16/1000001a669.000000a5/head//4, version: 397'2043, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(161a5b16/1000001a669.000000a5/head//4@397'2043, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9435> 2014-11-08 18:14:30.140214 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,1]) -9434> 2014-11-08 18:14:30.140248 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2371aee0 con 0x1cae2940 -9433> 2014-11-08 18:14:30.140272 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2371f620 con 0x1cae2680 -9432> 2014-11-08 18:14:30.140281 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x23719e00 con 0x1c5c02c0 -9431> 2014-11-08 18:14:30.140290 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2371a760 con 0x1cae0160 -9430> 2014-11-08 18:14:30.140299 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2371ddc0 con 0x1cae0f20 -9429> 2014-11-08 18:14:30.140308 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x237190e0 con 0x1cae0c60 -9428> 2014-11-08 18:14:30.140317 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x237192c0 con 0x1cae23c0 -9427> 2014-11-08 18:14:30.140326 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x23718f00 con 0x1cae56a0 -9426> 2014-11-08 18:14:30.140334 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x23719fe0 con 0x1cbf9ce0 -9425> 2014-11-08 18:14:30.140344 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2290a1c0 con 0x1d603020 -9424> 2014-11-08 18:14:30.140353 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x22908d20 con 0x1dbd9b80 -9423> 2014-11-08 18:14:30.140361 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2290c1a0 con 0x1d235d80 -9422> 2014-11-08 18:14:30.140370 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x21223a20 con 0x1cae1fa0 -9421> 2014-11-08 18:14:30.140379 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x259f7080 con 0x1cae1ce0 -9420> 2014-11-08 18:14:30.140388 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x1ec20b40 con 0x1cae1340 -9419> 2014-11-08 18:14:30.140397 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x2583da00 con 0x1cae1080 -9418> 2014-11-08 18:14:30.140406 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x22e9bc00 con 0x1cae1a20 -9417> 2014-11-08 18:14:30.140414 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:30.140244) v2 -- ?+0 0x225790e0 con 0x1cae2c00 -9416> 2014-11-08 18:14:30.141363 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1f4c30c0 con 0x1cae2680 -9415> 2014-11-08 18:14:30.141390 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x2583eea0 con 0x1cae56a0 -9414> 2014-11-08 18:14:30.141400 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x22e981e0 con 0x1cae0f20 -9413> 2014-11-08 18:14:30.141409 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x23d76360 con 0x1cae1ce0 -9412> 2014-11-08 18:14:30.141419 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1d4f2760 con 0x1cae2940 -9411> 2014-11-08 18:14:30.141613 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 302 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1d324560 con 0x1d235d80 -9410> 2014-11-08 18:14:30.141626 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x257c6ea0 con 0x1c5c02c0 -9409> 2014-11-08 18:14:30.141636 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x246ca760 con 0x1cae0160 -9408> 2014-11-08 18:14:30.141644 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1e894ec0 con 0x1cae0c60 -9407> 2014-11-08 18:14:30.141652 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1e536ea0 con 0x1cae1fa0 -9406> 2014-11-08 18:14:30.141660 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x2515eae0 con 0x1cae23c0 -9405> 2014-11-08 18:14:30.141668 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 495 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1eae3a20 con 0x1cbf9ce0 -9404> 2014-11-08 18:14:30.141677 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 495 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1dbfd820 con 0x1d603020 -9403> 2014-11-08 18:14:30.141691 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 302 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x24886720 con 0x1dbd9b80 -9402> 2014-11-08 18:14:30.141787 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22453de0 con 0x1c5c0580 -9401> 2014-11-08 18:14:30.141809 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(161a5b16/1000001a669.000000a5/head//4)]) v2 -- ?+0 0x1d5b3200 con 0x1c5c0580 -9400> 2014-11-08 18:14:30.143947 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1dec9680 con 0x1cae1080 -9399> 2014-11-08 18:14:30.143965 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x2234e720 con 0x1cae1a20 -9398> 2014-11-08 18:14:30.143975 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x1dbfe720 con 0x1cae2c00 -9397> 2014-11-08 18:14:30.143984 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3715 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.140244) v2 ==== 47+0+0 (754889297 0 0) 0x22452ee0 con 0x1cae1340 -9396> 2014-11-08 18:14:30.144626 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8441 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) v3 ==== 757+0+0 (1118691511 0 0) 0x1d33e300 con 0x1c5c0580 -9395> 2014-11-08 18:14:30.144639 7fc5a152e700 5 -- op tracker -- seq: 46707, time: 2014-11-08 18:14:30.144553, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9394> 2014-11-08 18:14:30.144644 7fc5a152e700 5 -- op tracker -- seq: 46707, time: 2014-11-08 18:14:30.144555, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9393> 2014-11-08 18:14:30.144650 7fc5a152e700 5 -- op tracker -- seq: 46707, time: 2014-11-08 18:14:30.144622, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9392> 2014-11-08 18:14:30.144656 7fc5a152e700 5 -- op tracker -- seq: 46707, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9391> 2014-11-08 18:14:30.144688 7fc5ad310700 5 -- op tracker -- seq: 46707, time: 2014-11-08 18:14:30.144688, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9390> 2014-11-08 18:14:30.144708 7fc5ad310700 5 -- op tracker -- seq: 46707, time: 2014-11-08 18:14:30.144708, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9389> 2014-11-08 18:14:30.144754 7fc5ad310700 5 -- op tracker -- seq: 46707, time: 2014-11-08 18:14:30.144754, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 161a5b16/1000001a669.000000a5/head//4) -9388> 2014-11-08 18:14:30.194735 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1566 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) v3 ==== 757+0+0 (241152211 0 0) 0x1d654800 con 0x21468c60 -9387> 2014-11-08 18:14:30.194770 7fc5a39d2700 5 -- op tracker -- seq: 46708, time: 2014-11-08 18:14:30.194645, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9386> 2014-11-08 18:14:30.194780 7fc5a39d2700 5 -- op tracker -- seq: 46708, time: 2014-11-08 18:14:30.194647, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9385> 2014-11-08 18:14:30.194784 7fc5a39d2700 5 -- op tracker -- seq: 46708, time: 2014-11-08 18:14:30.194728, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9384> 2014-11-08 18:14:30.194789 7fc5a39d2700 5 -- op tracker -- seq: 46708, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9383> 2014-11-08 18:14:30.194837 7fc5af314700 5 -- op tracker -- seq: 46708, time: 2014-11-08 18:14:30.194837, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9382> 2014-11-08 18:14:30.194850 7fc5af314700 5 -- op tracker -- seq: 46708, time: 2014-11-08 18:14:30.194850, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9381> 2014-11-08 18:14:30.194897 7fc5af314700 5 -- op tracker -- seq: 46708, time: 2014-11-08 18:14:30.194897, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6ca7bddd/10000019741.0000000b/head//4) -9380> 2014-11-08 18:14:30.340877 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3701 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:30.338735) v2 ==== 47+0+0 (3527542907 0 0) 0x21210000 con 0x1c8b7d20 -9379> 2014-11-08 18:14:30.340911 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.338735) v2 -- ?+0 0x21215a00 con 0x1c8b7d20 -9378> 2014-11-08 18:14:30.340942 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3701 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:30.338735) v2 ==== 47+0+0 (3527542907 0 0) 0x1d042d00 con 0x1c9f8580 -9377> 2014-11-08 18:14:30.340952 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.338735) v2 -- ?+0 0x2026ad00 con 0x1c9f8580 -9376> 2014-11-08 18:14:30.407718 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59222, attrs_read=0)) v1 -- ?+0 0x21838280 con 0x21468c60 -9375> 2014-11-08 18:14:30.407760 7fc5acb0f700 5 -- op tracker -- seq: 46686, time: 2014-11-08 18:14:30.407759, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59222, to_read={902d314e/1000002a575.0000003c/head//4=0,4194304}, attrs_to_read=)) -9374> 2014-11-08 18:14:30.440285 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8442 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) v2 ==== 110+0+0 (494001216 0 0) 0x1d5b3200 con 0x1c5c0580 -9373> 2014-11-08 18:14:30.440322 7fc5a152e700 5 -- op tracker -- seq: 46709, time: 2014-11-08 18:14:30.440186, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) -9372> 2014-11-08 18:14:30.440337 7fc5a152e700 5 -- op tracker -- seq: 46709, time: 2014-11-08 18:14:30.440188, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) -9371> 2014-11-08 18:14:30.440346 7fc5a152e700 5 -- op tracker -- seq: 46709, time: 2014-11-08 18:14:30.440264, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) -9370> 2014-11-08 18:14:30.440356 7fc5a152e700 5 -- op tracker -- seq: 46709, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) -9369> 2014-11-08 18:14:30.440398 7fc5afb15700 5 -- op tracker -- seq: 46709, time: 2014-11-08 18:14:30.440398, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) -9368> 2014-11-08 18:14:30.440737 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=a969e343/1000002baa9.00000005/head//4)) v1 -- ?+0 0x2562d7c0 con 0x1e50aec0 -9367> 2014-11-08 18:14:30.440755 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x25629440 con 0x42c86e0 -9366> 2014-11-08 18:14:30.440765 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is c899e343/10000025206.00000000/head//4 -9365> 2014-11-08 18:14:30.440803 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb 7fd8e343/10000028cf9.00000005/head//4) v3 -- ?+0 0x2482de80 con 0x1c5c0580 -9364> 2014-11-08 18:14:30.440924 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x25629440 con 0x42c86e0 -9363> 2014-11-08 18:14:30.440934 7fc5b9328700 5 -- op tracker -- seq: 46710, time: 2014-11-08 18:14:30.440761, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) -9362> 2014-11-08 18:14:30.440938 7fc5b9328700 5 -- op tracker -- seq: 46710, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) -9361> 2014-11-08 18:14:30.440941 7fc5b9328700 5 -- op tracker -- seq: 46710, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) -9360> 2014-11-08 18:14:30.440943 7fc5b9328700 5 -- op tracker -- seq: 46710, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) -9359> 2014-11-08 18:14:30.441020 7fc5ad310700 5 -- op tracker -- seq: 46710, time: 2014-11-08 18:14:30.441020, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) -9358> 2014-11-08 18:14:30.441369 7fc5a152e700 5 -- op tracker -- seq: 46709, time: 2014-11-08 18:14:30.441369, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(7fd8e343/10000028cf9.00000005/head//4)]) -9357> 2014-11-08 18:14:30.515317 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2149 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) v2 ==== 110+0+0 (1403590289 0 0) 0x2393ac00 con 0x1e50aec0 -9356> 2014-11-08 18:14:30.515341 7fc5a56ef700 5 -- op tracker -- seq: 46711, time: 2014-11-08 18:14:30.515233, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9355> 2014-11-08 18:14:30.515348 7fc5a56ef700 5 -- op tracker -- seq: 46711, time: 2014-11-08 18:14:30.515235, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9354> 2014-11-08 18:14:30.515351 7fc5a56ef700 5 -- op tracker -- seq: 46711, time: 2014-11-08 18:14:30.515311, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9353> 2014-11-08 18:14:30.515354 7fc5a56ef700 5 -- op tracker -- seq: 46711, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9352> 2014-11-08 18:14:30.515407 7fc5aeb13700 5 -- op tracker -- seq: 46711, time: 2014-11-08 18:14:30.515407, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9351> 2014-11-08 18:14:30.515484 7fc5aeb13700 5 -- op tracker -- seq: 46711, time: 2014-11-08 18:14:30.515483, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(2cc5dd13/10000019a83.0000003d/head//4)]) -9350> 2014-11-08 18:14:30.522884 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) v1 -- ?+0 0x21a4d500 con 0x42c86e0 -9349> 2014-11-08 18:14:30.522920 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4d500 con 0x42c86e0 -9348> 2014-11-08 18:14:30.522935 7fc5b9328700 5 -- op tracker -- seq: 46712, time: 2014-11-08 18:14:30.522909, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) -9347> 2014-11-08 18:14:30.522939 7fc5b9328700 5 -- op tracker -- seq: 46712, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) -9346> 2014-11-08 18:14:30.522941 7fc5b9328700 5 -- op tracker -- seq: 46712, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) -9345> 2014-11-08 18:14:30.522943 7fc5b9328700 5 -- op tracker -- seq: 46712, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) -9344> 2014-11-08 18:14:30.522978 7fc5afb15700 5 -- op tracker -- seq: 46712, time: 2014-11-08 18:14:30.522978, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) -9343> 2014-11-08 18:14:30.523006 7fc5afb15700 5 -- op tracker -- seq: 46712, time: 2014-11-08 18:14:30.523006, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=0)) -9342> 2014-11-08 18:14:30.522921 7fc5ad310700 5 -- op tracker -- seq: 46710, time: 2014-11-08 18:14:30.522920, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4385, to_read={a969e343/1000002baa9.00000005/head//4=0,4194304}, attrs_to_read=)) -9341> 2014-11-08 18:14:30.532969 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2150 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) v2 ==== 370+0+0 (1307864465 0 0) 0x1d5b5200 con 0x1e50aec0 -9340> 2014-11-08 18:14:30.532992 7fc5a56ef700 5 -- op tracker -- seq: 46713, time: 2014-11-08 18:14:30.532853, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) -9339> 2014-11-08 18:14:30.533001 7fc5a56ef700 5 -- op tracker -- seq: 46713, time: 2014-11-08 18:14:30.532855, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) -9338> 2014-11-08 18:14:30.533007 7fc5a56ef700 5 -- op tracker -- seq: 46713, time: 2014-11-08 18:14:30.532964, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) -9337> 2014-11-08 18:14:30.533012 7fc5a56ef700 5 -- op tracker -- seq: 46713, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) -9336> 2014-11-08 18:14:30.533044 7fc5ac30e700 5 -- op tracker -- seq: 46713, time: 2014-11-08 18:14:30.533044, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) -9335> 2014-11-08 18:14:30.533222 7fc5ac30e700 5 -- op tracker -- seq: 46713, time: 2014-11-08 18:14:30.533222, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(f957f418/1000001aaa8.0000004d/head//4),PushReplyOp(3558f418/10000019da8.00000026/head//4),PushReplyOp(4888f418/1000002a842.0000002c/head//4),PushReplyOp(1fa9f418/10000019b1e.000002bf/head//4),PushReplyOp(e4daf418/1000002b5ea.00000311/head//4)]) -9334> 2014-11-08 18:14:30.545081 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=9356dd13/1000001aa1e.00000034/head//4)) v1 -- ?+0 0x256286c0 con 0x1e50aec0 -9333> 2014-11-08 18:14:30.545112 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2562f2c0 con 0x42c86e0 -9332> 2014-11-08 18:14:30.545125 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=1 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is b1b7dd13/10000019aef.00000427/head//4 -9331> 2014-11-08 18:14:30.545151 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb 2cc5dd13/10000019a83.0000003d/head//4) v3 -- ?+0 0x1e802880 con 0x1c5c0580 -9330> 2014-11-08 18:14:30.545183 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2562f2c0 con 0x42c86e0 -9329> 2014-11-08 18:14:30.545195 7fc5b9328700 5 -- op tracker -- seq: 46714, time: 2014-11-08 18:14:30.545121, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) -9328> 2014-11-08 18:14:30.545203 7fc5b9328700 5 -- op tracker -- seq: 46714, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) -9327> 2014-11-08 18:14:30.545206 7fc5b9328700 5 -- op tracker -- seq: 46714, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) -9326> 2014-11-08 18:14:30.545208 7fc5b9328700 5 -- op tracker -- seq: 46714, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) -9325> 2014-11-08 18:14:30.545289 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb 2cc5dd13/10000019a83.0000003d/head//4) v3 -- ?+0 0x2482e780 con 0x1e50aec0 -9324> 2014-11-08 18:14:30.545593 7fc5aeb13700 5 -- op tracker -- seq: 46714, time: 2014-11-08 18:14:30.545593, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) -9323> 2014-11-08 18:14:30.633790 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) v1 -- ?+0 0x21a4e400 con 0x42c86e0 -9322> 2014-11-08 18:14:30.633839 7fc5aeb13700 5 -- op tracker -- seq: 46714, time: 2014-11-08 18:14:30.633839, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4386, to_read={9356dd13/1000001aa1e.00000034/head//4=0,4194304}, attrs_to_read=)) -9321> 2014-11-08 18:14:30.633841 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4e400 con 0x42c86e0 -9320> 2014-11-08 18:14:30.633868 7fc5b9328700 5 -- op tracker -- seq: 46715, time: 2014-11-08 18:14:30.633827, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) -9319> 2014-11-08 18:14:30.633873 7fc5b9328700 5 -- op tracker -- seq: 46715, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) -9318> 2014-11-08 18:14:30.633875 7fc5b9328700 5 -- op tracker -- seq: 46715, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) -9317> 2014-11-08 18:14:30.633876 7fc5b9328700 5 -- op tracker -- seq: 46715, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) -9316> 2014-11-08 18:14:30.633907 7fc5ac30e700 5 -- op tracker -- seq: 46715, time: 2014-11-08 18:14:30.633907, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) -9315> 2014-11-08 18:14:30.633952 7fc5ac30e700 5 -- op tracker -- seq: 46715, time: 2014-11-08 18:14:30.633952, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=0)) -9314> 2014-11-08 18:14:30.638547 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=472bf418/10000029051.0000000f/head//4,fe9bf418/1000001a0b9.0000068e/head//4,af1cf418/1000002b54f.0000016e/head//4,b1bcf418/1000001ac2f.0000004f/head//4,700df418/10000025f2d.00000000/head//4)) v1 -- ?+0 0x2562a640 con 0x1c5c0580 -9313> 2014-11-08 18:14:30.638574 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2562aac0 con 0x42c86e0 -9312> 2014-11-08 18:14:30.638590 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is ca5df418/1000001b502.00000378/head//4 -9311> 2014-11-08 18:14:30.638641 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb e4daf418/1000002b5ea.00000311/head//4) v3 -- ?+0 0x22761f80 con 0x1e50aec0 -9310> 2014-11-08 18:14:30.638795 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2562aac0 con 0x42c86e0 -9309> 2014-11-08 18:14:30.638810 7fc5b9328700 5 -- op tracker -- seq: 46716, time: 2014-11-08 18:14:30.638585, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) -9308> 2014-11-08 18:14:30.638820 7fc5b9328700 5 -- op tracker -- seq: 46716, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) -9307> 2014-11-08 18:14:30.638826 7fc5b9328700 5 -- op tracker -- seq: 46716, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) -9306> 2014-11-08 18:14:30.638833 7fc5b9328700 5 -- op tracker -- seq: 46716, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) -9305> 2014-11-08 18:14:30.638856 7fc5aeb13700 5 -- op tracker -- seq: 46716, time: 2014-11-08 18:14:30.638856, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) -9304> 2014-11-08 18:14:30.670197 7fc5c362f700 5 osd.9 5773 tick -9303> 2014-11-08 18:14:30.670329 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -9302> 2014-11-08 18:14:30.670343 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(3 pgs tid 1324 v 0) v1 -- ?+0 0x22d7f0c0 con 0x44ed6a0 -9301> 2014-11-08 18:14:30.676231 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1567 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (253734370 0 0) 0x22879b00 con 0x21468c60 -9300> 2014-11-08 18:14:30.676267 7fc5a39d2700 5 -- op tracker -- seq: 46717, time: 2014-11-08 18:14:30.676155, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) -9299> 2014-11-08 18:14:30.676275 7fc5a39d2700 5 -- op tracker -- seq: 46717, time: 2014-11-08 18:14:30.676158, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) -9298> 2014-11-08 18:14:30.676278 7fc5a39d2700 5 -- op tracker -- seq: 46717, time: 2014-11-08 18:14:30.676225, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) -9297> 2014-11-08 18:14:30.676281 7fc5a39d2700 5 -- op tracker -- seq: 46717, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) -9296> 2014-11-08 18:14:30.676327 7fc5af314700 5 -- op tracker -- seq: 46717, time: 2014-11-08 18:14:30.676327, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) -9295> 2014-11-08 18:14:30.757866 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8443 ==== MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/ 1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (1496265523 0 0) 0x1d5b3200 con 0x1c5c0580 -9294> 2014-11-08 18:14:30.757933 7fc5a152e700 5 -- op tracker -- seq: 46718, time: 2014-11-08 18:14:30.522356, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9293> 2014-11-08 18:14:30.757964 7fc5a152e700 5 -- op tracker -- seq: 46718, time: 2014-11-08 18:14:30.522358, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9292> 2014-11-08 18:14:30.757989 7fc5a152e700 5 -- op tracker -- seq: 46718, time: 2014-11-08 18:14:30.757829, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9291> 2014-11-08 18:14:30.758013 7fc5a152e700 5 -- op tracker -- seq: 46718, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/ 1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9290> 2014-11-08 18:14:30.758081 7fc5adb11700 5 -- op tracker -- seq: 46718, time: 2014-11-08 18:14:30.758081, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9289> 2014-11-08 18:14:30.758265 7fc5adb11700 5 -- op tracker -- seq: 46718, time: 2014-11-08 18:14:30.758265, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(d882fcb5/1000002aa68.000000aa/head//4, version: 3458'5696, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d882fcb5/1000002aa68.000000aa/head//4@3458'5696, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c2fcb5/1000001a9ac.0000016e/head//4, version: 2829'2372, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c2fcb5/1000001a9ac.0000016e/head//4@2829'2372, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c593fcb5/1000002adf0.0000005b/head//4, version: 3468'6096, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c593fcb5/1000002adf0.0000005b/head//4@3468'6096, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1ce3fcb5/1000001b479.00000298/head//4, version: 3033'2990, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ce3fcb5/ 1000001b479.00000298/head//4@3033'2990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ee3fcb5/1000002b033.00000021/head//4, version: 4157'7747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ee3fcb5/1000002b033.00000021/head//4@4157'7747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9288> 2014-11-08 18:14:30.781004 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59226, attrs_read=0)) v1 -- ?+0 0x224bc100 con 0x21468c60 -9287> 2014-11-08 18:14:30.781050 7fc5af314700 5 -- op tracker -- seq: 46717, time: 2014-11-08 18:14:30.781050, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59226, to_read={444d314e/1000002b87d.0000005e/head//4=0,4194304}, attrs_to_read=)) -9286> 2014-11-08 18:14:30.811940 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3730 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:30.811185) v2 ==== 47+0+0 (707115848 0 0) 0x23680d20 con 0x1cce3180 -9285> 2014-11-08 18:14:30.811968 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.811185) v2 -- ?+0 0x1d042d00 con 0x1cce3180 -9284> 2014-11-08 18:14:30.811992 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3730 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:30.811185) v2 ==== 47+0+0 (707115848 0 0) 0x2515ad00 con 0x1cce3700 -9283> 2014-11-08 18:14:30.812001 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:30.811185) v2 -- ?+0 0x21210000 con 0x1cce3700 -9282> 2014-11-08 18:14:30.923481 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8444 ==== MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 11453+0+0 (170721802 0 0) 0x1d3aba00 con 0x1c5c0580 -9281> 2014-11-08 18:14:30.923525 7fc5a152e700 5 -- op tracker -- seq: 46719, time: 2014-11-08 18:14:30.923132, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9280> 2014-11-08 18:14:30.923540 7fc5a152e700 5 -- op tracker -- seq: 46719, time: 2014-11-08 18:14:30.923136, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9279> 2014-11-08 18:14:30.923548 7fc5a152e700 5 -- op tracker -- seq: 46719, time: 2014-11-08 18:14:30.923471, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9278> 2014-11-08 18:14:30.923555 7fc5a152e700 5 -- op tracker -- seq: 46719, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9277> 2014-11-08 18:14:30.923622 7fc5afb15700 5 -- op tracker -- seq: 46719, time: 2014-11-08 18:14:30.923622, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9276> 2014-11-08 18:14:30.923716 7fc5afb15700 5 -- op tracker -- seq: 46719, time: 2014-11-08 18:14:30.923716, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(f5a5b16/1000001c728.00000000/head//4, version: 3033'3363, data_included: [0~10240], data_size: 10240, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(f5a5b16/1000001c728.00000000/head//4@3033'3363, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:20480, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9275> 2014-11-08 18:14:30.987087 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) v1 -- ?+0 0x21a4cb00 con 0x42c86e0 -9274> 2014-11-08 18:14:30.987136 7fc5aeb13700 5 -- op tracker -- seq: 46716, time: 2014-11-08 18:14:30.987136, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4387, to_read={472bf418/10000029051.0000000f/head//4=0,4194304,fe9bf418/1000001a0b9.0000068e/head//4=0,4194304,af1cf418/1000002b54f.0000016e/head//4=0,4194304,b1bcf418/1000001ac2f.0000004f/head//4=0,4194304,700df418/10000025f2d.00000000/head//4=0,4194304}, attrs_to_read=)) -9273> 2014-11-08 18:14:30.987195 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4cb00 con 0x42c86e0 -9272> 2014-11-08 18:14:30.987212 7fc5b9328700 5 -- op tracker -- seq: 46720, time: 2014-11-08 18:14:30.987123, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) -9271> 2014-11-08 18:14:30.987217 7fc5b9328700 5 -- op tracker -- seq: 46720, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) -9270> 2014-11-08 18:14:30.987221 7fc5b9328700 5 -- op tracker -- seq: 46720, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) -9269> 2014-11-08 18:14:30.987225 7fc5b9328700 5 -- op tracker -- seq: 46720, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) -9268> 2014-11-08 18:14:30.987282 7fc5ac30e700 5 -- op tracker -- seq: 46720, time: 2014-11-08 18:14:30.987282, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) -9267> 2014-11-08 18:14:30.987346 7fc5ac30e700 5 -- op tracker -- seq: 46720, time: 2014-11-08 18:14:30.987346, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=0)) -9266> 2014-11-08 18:14:31.047424 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22455dc0 con 0x1c5c0580 -9265> 2014-11-08 18:14:31.047466 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22450000 con 0x1c5c0580 -9264> 2014-11-08 18:14:31.047473 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x224514a0 con 0x1c5c0580 -9263> 2014-11-08 18:14:31.047479 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22450f00 con 0x1c5c0580 -9262> 2014-11-08 18:14:31.047485 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22452d00 con 0x1c5c0580 -9261> 2014-11-08 18:14:31.047493 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(d882fcb5/1000002aa68.000000aa/head//4),PushReplyOp(d3c2fcb5/1000001a9ac.0000016e/head//4),PushReplyOp(c593fcb5/1000002adf0.0000005b/head//4),PushReplyOp(1ce3fcb5/1000001b479.00000298/head//4),PushReplyOp(4ee3fcb5/1000002b033.00000021/head//4)]) v2 -- ?+0 0x2393ae00 con 0x1c5c0580 -9260> 2014-11-08 18:14:31.050918 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8445 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) v3 ==== 757+0+0 (836889511 0 0) 0x2482de80 con 0x1c5c0580 -9259> 2014-11-08 18:14:31.050938 7fc5a152e700 5 -- op tracker -- seq: 46721, time: 2014-11-08 18:14:31.050864, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9258> 2014-11-08 18:14:31.050949 7fc5a152e700 5 -- op tracker -- seq: 46721, time: 2014-11-08 18:14:31.050867, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9257> 2014-11-08 18:14:31.050954 7fc5a152e700 5 -- op tracker -- seq: 46721, time: 2014-11-08 18:14:31.050912, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9256> 2014-11-08 18:14:31.050959 7fc5a152e700 5 -- op tracker -- seq: 46721, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9255> 2014-11-08 18:14:31.050997 7fc5b0316700 5 -- op tracker -- seq: 46721, time: 2014-11-08 18:14:31.050997, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9254> 2014-11-08 18:14:31.051007 7fc5b0316700 5 -- op tracker -- seq: 46721, time: 2014-11-08 18:14:31.051007, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9253> 2014-11-08 18:14:31.051069 7fc5a152e700 5 -- op tracker -- seq: 46721, time: 2014-11-08 18:14:31.051069, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 4ee3fcb5/1000002b033.00000021/head//4) -9252> 2014-11-08 18:14:31.055336 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1568 ==== MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f8abddd/ 1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6991289+0+0 (104556894 0 0) 0x1d5b1000 con 0x21468c60 -9251> 2014-11-08 18:14:31.055384 7fc5a39d2700 5 -- op tracker -- seq: 46722, time: 2014-11-08 18:14:30.944568, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6f8abddd/1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9250> 2014-11-08 18:14:31.055405 7fc5a39d2700 5 -- op tracker -- seq: 46722, time: 2014-11-08 18:14:30.944577, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6f8abddd/1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9249> 2014-11-08 18:14:31.055420 7fc5a39d2700 5 -- op tracker -- seq: 46722, time: 2014-11-08 18:14:31.055326, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6f8abddd/1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9248> 2014-11-08 18:14:31.055436 7fc5a39d2700 5 -- op tracker -- seq: 46722, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f8abddd/ 1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9247> 2014-11-08 18:14:31.055484 7fc5acb0f700 5 -- op tracker -- seq: 46722, time: 2014-11-08 18:14:31.055484, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 6f8abddd/1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9246> 2014-11-08 18:14:31.055641 7fc5acb0f700 5 -- op tracker -- seq: 46722, time: 2014-11-08 18:14:31.055641, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(7b68bddd/10000029740.00000084/head//4, version: 3033'8418, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b68bddd/10000029740.00000084/head//4@3033'8418, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42c9bddd/1000002bb04.0000006d/head//4, version: 3473'14473, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42c9bddd/1000002bb04.0000006d/head//4@3473'14473, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(671abddd/10000024bc0.00000001/head//4, version: 3033'6829, data_included: [0~696320], data_size: 696320, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(671abddd/10000024bc0.00000001/head//4@3033'6829, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1392640, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6f8abddd/1000002a98a.0000012d/head//4, version: 3458'11070, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6f8abddd/ 1000002a98a.0000012d/head//4@3458'11070, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9245> 2014-11-08 18:14:31.101153 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25fc1860 con 0x1c5c0580 -9244> 2014-11-08 18:14:31.101199 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(f5a5b16/1000001c728.00000000/head//4)]) v2 -- ?+0 0x1d5b2600 con 0x1c5c0580 -9243> 2014-11-08 18:14:31.200070 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2151 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) v1 ==== 2097696+0+0 (1439443986 0 0) 0x21a49900 con 0x1e50aec0 -9242> 2014-11-08 18:14:31.200105 7fc5a56ef700 5 -- op tracker -- seq: 46723, time: 2014-11-08 18:14:31.143152, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) -9241> 2014-11-08 18:14:31.200115 7fc5a56ef700 5 -- op tracker -- seq: 46723, time: 2014-11-08 18:14:31.143158, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) -9240> 2014-11-08 18:14:31.200118 7fc5a56ef700 5 -- op tracker -- seq: 46723, time: 2014-11-08 18:14:31.200060, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) -9239> 2014-11-08 18:14:31.200122 7fc5a56ef700 5 -- op tracker -- seq: 46723, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) -9238> 2014-11-08 18:14:31.200160 7fc5aeb13700 5 -- op tracker -- seq: 46723, time: 2014-11-08 18:14:31.200160, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) -9237> 2014-11-08 18:14:31.200198 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2152 ==== MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) v2 ==== 175+0+0 (1060502966 0 0) 0x1c6a4000 con 0x1e50aec0 -9236> 2014-11-08 18:14:31.200209 7fc5a56ef700 5 -- op tracker -- seq: 46724, time: 2014-11-08 18:14:31.200152, event: header_read, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) -9235> 2014-11-08 18:14:31.200213 7fc5a56ef700 5 -- op tracker -- seq: 46724, time: 2014-11-08 18:14:31.200153, event: throttled, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) -9234> 2014-11-08 18:14:31.200217 7fc5a56ef700 5 -- op tracker -- seq: 46724, time: 2014-11-08 18:14:31.200195, event: all_read, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) -9233> 2014-11-08 18:14:31.200220 7fc5a56ef700 5 -- op tracker -- seq: 46724, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) -9232> 2014-11-08 18:14:31.200245 7fc5adb11700 5 -- op tracker -- seq: 46724, time: 2014-11-08 18:14:31.200245, event: reached_pg, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) -9231> 2014-11-08 18:14:31.200389 7fc5a56ef700 5 -- op tracker -- seq: 46724, time: 2014-11-08 18:14:31.200389, event: done, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(aa4a231a/100000283fc.00000036/head//4),PushReplyOp(b45a231a/1000001a7b0.0000003f/head//4)]) -9230> 2014-11-08 18:14:31.205373 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(9356dd13/1000001aa1e.00000034/head//4, version: 2829'2345, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9356dd13/1000001aa1e.00000034/head//4@2829'2345, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393ac00 con 0x1c5c0580 -9229> 2014-11-08 18:14:31.205430 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(9356dd13/1000001aa1e.00000034/head//4, version: 2829'2345, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9356dd13/1000001aa1e.00000034/head//4@2829'2345, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393dc00 con 0x1e50aec0 -9228> 2014-11-08 18:14:31.210423 7fc5aeb13700 5 -- op tracker -- seq: 46723, time: 2014-11-08 18:14:31.210422, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4386, attrs_read=1)) -9227> 2014-11-08 18:14:31.262565 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.1as2 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=891b231a/100000199b7.00000367/head//4,e9ab231a/1000002a767.00000093/head//4)) v1 -- ?+0 0x2562ca40 con 0x1c5c0580 -9226> 2014-11-08 18:14:31.262604 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x25629b00 con 0x42c86e0 -9225> 2014-11-08 18:14:31.262618 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.1as0( v 5734'8347 (3458'5305,5734'8347] local-les=5760 n=7730 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=2 bft=5(1) crt=5734'8342 lcod 5734'8346 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is ab0c231a/1000001a0eb.00000080/head//4 -9224> 2014-11-08 18:14:31.262646 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.1as1 e 5773/5773 lb b45a231a/1000001a7b0.0000003f/head//4) v3 -- ?+0 0x22762880 con 0x1e50aec0 -9223> 2014-11-08 18:14:31.262679 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x25629b00 con 0x42c86e0 -9222> 2014-11-08 18:14:31.262712 7fc5b9328700 5 -- op tracker -- seq: 46725, time: 2014-11-08 18:14:31.262613, event: header_read, op: MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) -9221> 2014-11-08 18:14:31.262723 7fc5b9328700 5 -- op tracker -- seq: 46725, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) -9220> 2014-11-08 18:14:31.262728 7fc5b9328700 5 -- op tracker -- seq: 46725, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) -9219> 2014-11-08 18:14:31.262733 7fc5b9328700 5 -- op tracker -- seq: 46725, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) -9218> 2014-11-08 18:14:31.262765 7fc5b0316700 5 -- op tracker -- seq: 46725, time: 2014-11-08 18:14:31.262765, event: reached_pg, op: MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) -9217> 2014-11-08 18:14:31.266247 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8446 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) v1 ==== 10234524+0+0 (3258327750 0 0) 0x1c898000 con 0x1c5c0580 -9216> 2014-11-08 18:14:31.266277 7fc5a152e700 5 -- op tracker -- seq: 46726, time: 2014-11-08 18:14:31.075941, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) -9215> 2014-11-08 18:14:31.266286 7fc5a152e700 5 -- op tracker -- seq: 46726, time: 2014-11-08 18:14:31.075945, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) -9214> 2014-11-08 18:14:31.266289 7fc5a152e700 5 -- op tracker -- seq: 46726, time: 2014-11-08 18:14:31.266238, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) -9213> 2014-11-08 18:14:31.266294 7fc5a152e700 5 -- op tracker -- seq: 46726, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) -9212> 2014-11-08 18:14:31.266329 7fc5ac30e700 5 -- op tracker -- seq: 46726, time: 2014-11-08 18:14:31.266329, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) -9211> 2014-11-08 18:14:31.266390 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8447 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) v3 ==== 757+0+0 (3436898252 0 0) 0x2482de80 con 0x1c5c0580 -9210> 2014-11-08 18:14:31.266399 7fc5a152e700 5 -- op tracker -- seq: 46727, time: 2014-11-08 18:14:31.266355, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9209> 2014-11-08 18:14:31.266402 7fc5a152e700 5 -- op tracker -- seq: 46727, time: 2014-11-08 18:14:31.266356, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9208> 2014-11-08 18:14:31.266406 7fc5a152e700 5 -- op tracker -- seq: 46727, time: 2014-11-08 18:14:31.266386, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9207> 2014-11-08 18:14:31.266408 7fc5a152e700 5 -- op tracker -- seq: 46727, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9206> 2014-11-08 18:14:31.266714 7fc5ad310700 5 -- op tracker -- seq: 46727, time: 2014-11-08 18:14:31.266714, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9205> 2014-11-08 18:14:31.266723 7fc5ad310700 5 -- op tracker -- seq: 46727, time: 2014-11-08 18:14:31.266723, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9204> 2014-11-08 18:14:31.266767 7fc5ad310700 5 -- op tracker -- seq: 46727, time: 2014-11-08 18:14:31.266767, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f5a5b16/1000001c728.00000000/head//4) -9203> 2014-11-08 18:14:31.282041 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8448 ==== MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 1115024+0+0 (2095565629 0 0) 0x1d5b2600 con 0x1c5c0580 -9202> 2014-11-08 18:14:31.282093 7fc5a152e700 5 -- op tracker -- seq: 46728, time: 2014-11-08 18:14:31.266424, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9201> 2014-11-08 18:14:31.282112 7fc5a152e700 5 -- op tracker -- seq: 46728, time: 2014-11-08 18:14:31.266425, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9200> 2014-11-08 18:14:31.282125 7fc5a152e700 5 -- op tracker -- seq: 46728, time: 2014-11-08 18:14:31.282013, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9199> 2014-11-08 18:14:31.282137 7fc5a152e700 5 -- op tracker -- seq: 46728, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9198> 2014-11-08 18:14:31.282183 7fc5adb11700 5 -- op tracker -- seq: 46728, time: 2014-11-08 18:14:31.282183, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9197> 2014-11-08 18:14:31.282289 7fc5adb11700 5 -- op tracker -- seq: 46728, time: 2014-11-08 18:14:31.282288, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(ebf3fcb5/1000001a9f2.0000022f/head//4, version: 2829'2438, data_included: [0~1114112], data_size: 1114112, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ebf3fcb5/1000001a9f2.0000022f/head//4@2829'2438, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2228224, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -9196> 2014-11-08 18:14:31.290817 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(472bf418/10000029051.0000000f/head//4, version: 3033'4104, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(472bf418/10000029051.0000000f/head//4@3033'4104, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe9bf418/1000001a0b9.0000068e/head//4, version: 397'1815, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe9bf418/1000001a0b9.0000068e/head//4@397'1815, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(af1cf418/1000002b54f.0000016e/head//4, version: 3473'6641, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(af1cf418/1000002b54f.0000016e/head//4@3473'6641, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b1bcf418/1000001ac2f.0000004f/head//4, version: 2829'2668, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b1bcf418/1000001ac2f.0000004f/ head//4@2829'2668, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(700df418/10000025f2d.00000000/head//4, version: 3033'3724, data_included: [0~1843200], data_size: 1843200, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(700df418/10000025f2d.00000000/head//4@3033'3724, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3686400, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d5b5200 con 0x1e50aec0 -9195> 2014-11-08 18:14:31.290905 7fc5ac30e700 5 -- op tracker -- seq: 46726, time: 2014-11-08 18:14:31.290905, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4387, attrs_read=5)) -9194> 2014-11-08 18:14:31.304468 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2153 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) v2 ==== 305+0+0 (343138742 0 0) 0x2393dc00 con 0x1e50aec0 -9193> 2014-11-08 18:14:31.304513 7fc5a56ef700 5 -- op tracker -- seq: 46729, time: 2014-11-08 18:14:31.304372, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) -9192> 2014-11-08 18:14:31.304526 7fc5a56ef700 5 -- op tracker -- seq: 46729, time: 2014-11-08 18:14:31.304376, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) -9191> 2014-11-08 18:14:31.304532 7fc5a56ef700 5 -- op tracker -- seq: 46729, time: 2014-11-08 18:14:31.304463, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) -9190> 2014-11-08 18:14:31.304537 7fc5a56ef700 5 -- op tracker -- seq: 46729, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) -9189> 2014-11-08 18:14:31.304594 7fc5adb11700 5 -- op tracker -- seq: 46729, time: 2014-11-08 18:14:31.304594, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) -9188> 2014-11-08 18:14:31.304768 7fc5adb11700 5 -- op tracker -- seq: 46729, time: 2014-11-08 18:14:31.304768, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(f68b1d83/1000001b477.000007ee/head//4),PushReplyOp(a21c1d83/1000001a103.00000095/head//4),PushReplyOp(42c1d83/1000001ab1a.00000015/head//4),PushReplyOp(b73c1d83/1000002a908.0000000a/head//4)]) -9187> 2014-11-08 18:14:31.305043 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=885c1d83/1000002b519.00000021/head//4)) v1 -- ?+0 0x2562d100 con 0x1c5c0580 -9186> 2014-11-08 18:14:31.305061 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x25628d80 con 0x42c86e0 -9185> 2014-11-08 18:14:31.305073 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is adfc1d83/1000002a6c6.00000008/head//4 -9184> 2014-11-08 18:14:31.305126 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb b73c1d83/1000002a908.0000000a/head//4) v3 -- ?+0 0x22764c80 con 0x1e50aec0 -9183> 2014-11-08 18:14:31.305217 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x25628d80 con 0x42c86e0 -9182> 2014-11-08 18:14:31.305229 7fc5b9328700 5 -- op tracker -- seq: 46730, time: 2014-11-08 18:14:31.305069, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) -9181> 2014-11-08 18:14:31.305235 7fc5b9328700 5 -- op tracker -- seq: 46730, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) -9180> 2014-11-08 18:14:31.305238 7fc5b9328700 5 -- op tracker -- seq: 46730, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) -9179> 2014-11-08 18:14:31.305242 7fc5b9328700 5 -- op tracker -- seq: 46730, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) -9178> 2014-11-08 18:14:31.305260 7fc5adb11700 5 -- op tracker -- seq: 46730, time: 2014-11-08 18:14:31.305260, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) -9177> 2014-11-08 18:14:31.374062 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12042 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3400884271 0 0) 0x1ce672c0 con 0x1c5c2680 -9176> 2014-11-08 18:14:31.374096 7fc5a6cfe700 5 -- op tracker -- seq: 46731, time: 2014-11-08 18:14:31.373966, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) -9175> 2014-11-08 18:14:31.374107 7fc5a6cfe700 5 -- op tracker -- seq: 46731, time: 2014-11-08 18:14:31.373971, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) -9174> 2014-11-08 18:14:31.374112 7fc5a6cfe700 5 -- op tracker -- seq: 46731, time: 2014-11-08 18:14:31.374053, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) -9173> 2014-11-08 18:14:31.374117 7fc5a6cfe700 5 -- op tracker -- seq: 46731, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) -9172> 2014-11-08 18:14:31.374279 7fc5aeb13700 5 -- op tracker -- seq: 46731, time: 2014-11-08 18:14:31.374279, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) -9171> 2014-11-08 18:14:31.376180 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12043 ==== MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (3695057718 0 0) 0x1ce65100 con 0x1c5c2680 -9170> 2014-11-08 18:14:31.376195 7fc5a6cfe700 5 -- op tracker -- seq: 46732, time: 2014-11-08 18:14:31.376120, event: header_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) -9169> 2014-11-08 18:14:31.376205 7fc5a6cfe700 5 -- op tracker -- seq: 46732, time: 2014-11-08 18:14:31.376121, event: throttled, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) -9168> 2014-11-08 18:14:31.376212 7fc5a6cfe700 5 -- op tracker -- seq: 46732, time: 2014-11-08 18:14:31.376177, event: all_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) -9167> 2014-11-08 18:14:31.376218 7fc5a6cfe700 5 -- op tracker -- seq: 46732, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) -9166> 2014-11-08 18:14:31.376245 7fc5afb15700 5 -- op tracker -- seq: 46732, time: 2014-11-08 18:14:31.376245, event: reached_pg, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) -9165> 2014-11-08 18:14:31.381765 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x25ed6900 con 0x21468c60 -9164> 2014-11-08 18:14:31.381804 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x25ed4ec0 con 0x21468c60 -9163> 2014-11-08 18:14:31.381812 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x25ed5280 con 0x21468c60 -9162> 2014-11-08 18:14:31.382343 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1d1b2b20 con 0x21468c60 -9161> 2014-11-08 18:14:31.382355 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(7b68bddd/10000029740.00000084/head//4),PushReplyOp(42c9bddd/1000002bb04.0000006d/head//4),PushReplyOp(671abddd/10000024bc0.00000001/head//4),PushReplyOp(6f8abddd/1000002a98a.0000012d/head//4)]) v2 -- ?+0 0x23938c00 con 0x21468c60 -9160> 2014-11-08 18:14:31.392945 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 490 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:31.391084) v2 ==== 47+0+0 (3072095664 0 0) 0x24885a00 con 0x1eb9c8e0 -9159> 2014-11-08 18:14:31.392982 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:31.391084) v2 -- ?+0 0x2515ad00 con 0x1eb9c8e0 -9158> 2014-11-08 18:14:31.402647 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 490 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:31.391084) v2 ==== 47+0+0 (3072095664 0 0) 0x22e5bde0 con 0x1eb9d800 -9157> 2014-11-08 18:14:31.402693 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:31.391084) v2 -- ?+0 0x23680d20 con 0x1eb9d800 -9156> 2014-11-08 18:14:31.414976 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) v1 -- ?+0 0x1c89a080 con 0x42c86e0 -9155> 2014-11-08 18:14:31.415022 7fc5b0316700 5 -- op tracker -- seq: 46725, time: 2014-11-08 18:14:31.415022, event: done, op: MOSDECSubOpRead(4.1as0 5773 ECSubRead(tid=4388, to_read={891b231a/100000199b7.00000367/head//4=0,4194304,e9ab231a/1000002a767.00000093/head//4=0,4194304}, attrs_to_read=)) -9154> 2014-11-08 18:14:31.415026 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1c89a080 con 0x42c86e0 -9153> 2014-11-08 18:14:31.415041 7fc5b9328700 5 -- op tracker -- seq: 46733, time: 2014-11-08 18:14:31.415010, event: header_read, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) -9152> 2014-11-08 18:14:31.415045 7fc5b9328700 5 -- op tracker -- seq: 46733, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) -9151> 2014-11-08 18:14:31.415047 7fc5b9328700 5 -- op tracker -- seq: 46733, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) -9150> 2014-11-08 18:14:31.415049 7fc5b9328700 5 -- op tracker -- seq: 46733, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) -9149> 2014-11-08 18:14:31.415080 7fc5b0316700 5 -- op tracker -- seq: 46733, time: 2014-11-08 18:14:31.415080, event: reached_pg, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) -9148> 2014-11-08 18:14:31.415121 7fc5b0316700 5 -- op tracker -- seq: 46733, time: 2014-11-08 18:14:31.415121, event: done, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=0)) -9147> 2014-11-08 18:14:31.445892 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1d1b3480 con 0x1c5c0580 -9146> 2014-11-08 18:14:31.445921 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(ebf3fcb5/1000001a9f2.0000022f/head//4)]) v2 -- ?+0 0x1d5b3200 con 0x1c5c0580 -9145> 2014-11-08 18:14:31.448329 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2154 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) v1 ==== 2097696+0+0 (2108047236 0 0) 0x21a4d000 con 0x1e50aec0 -9144> 2014-11-08 18:14:31.448364 7fc5a56ef700 5 -- op tracker -- seq: 46734, time: 2014-11-08 18:14:31.313632, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) -9143> 2014-11-08 18:14:31.448378 7fc5a56ef700 5 -- op tracker -- seq: 46734, time: 2014-11-08 18:14:31.313636, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) -9142> 2014-11-08 18:14:31.448387 7fc5a56ef700 5 -- op tracker -- seq: 46734, time: 2014-11-08 18:14:31.448320, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) -9141> 2014-11-08 18:14:31.448416 7fc5a56ef700 5 -- op tracker -- seq: 46734, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) -9140> 2014-11-08 18:14:31.448458 7fc5ad310700 5 -- op tracker -- seq: 46734, time: 2014-11-08 18:14:31.448458, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) -9139> 2014-11-08 18:14:31.452369 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2155 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) v3 ==== 757+0+0 (3737198943 0 0) 0x22762880 con 0x1e50aec0 -9138> 2014-11-08 18:14:31.452432 7fc5a56ef700 5 -- op tracker -- seq: 46735, time: 2014-11-08 18:14:31.452288, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9137> 2014-11-08 18:14:31.452447 7fc5a56ef700 5 -- op tracker -- seq: 46735, time: 2014-11-08 18:14:31.452290, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9136> 2014-11-08 18:14:31.452456 7fc5a56ef700 5 -- op tracker -- seq: 46735, time: 2014-11-08 18:14:31.452366, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9135> 2014-11-08 18:14:31.452465 7fc5a56ef700 5 -- op tracker -- seq: 46735, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9134> 2014-11-08 18:14:31.452607 7fc5af314700 5 -- op tracker -- seq: 46735, time: 2014-11-08 18:14:31.452607, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9133> 2014-11-08 18:14:31.452620 7fc5af314700 5 -- op tracker -- seq: 46735, time: 2014-11-08 18:14:31.452620, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9132> 2014-11-08 18:14:31.452646 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2156 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1decd0a0 con 0x1e50aec0 -9131> 2014-11-08 18:14:31.452660 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452498, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -9130> 2014-11-08 18:14:31.452664 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452498, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -9129> 2014-11-08 18:14:31.452667 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452629, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -9128> 2014-11-08 18:14:31.452668 7fc5af314700 5 -- op tracker -- seq: 46735, time: 2014-11-08 18:14:31.452667, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 88e990b7/1000001acc4.00000011/head//4) -9127> 2014-11-08 18:14:31.452670 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452657, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -9126> 2014-11-08 18:14:31.452673 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452673, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -9125> 2014-11-08 18:14:31.452678 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452678, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -9124> 2014-11-08 18:14:31.452682 7fc5b9b29700 5 -- op tracker -- seq: 46736, time: 2014-11-08 18:14:31.452682, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -9123> 2014-11-08 18:14:31.452735 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2157 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) v2 ==== 110+0+0 (2414930278 0 0) 0x1c6a4000 con 0x1e50aec0 -9122> 2014-11-08 18:14:31.452752 7fc5a56ef700 5 -- op tracker -- seq: 46737, time: 2014-11-08 18:14:31.452709, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9121> 2014-11-08 18:14:31.452762 7fc5a56ef700 5 -- op tracker -- seq: 46737, time: 2014-11-08 18:14:31.452710, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9120> 2014-11-08 18:14:31.452771 7fc5a56ef700 5 -- op tracker -- seq: 46737, time: 2014-11-08 18:14:31.452734, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9119> 2014-11-08 18:14:31.452780 7fc5a56ef700 5 -- op tracker -- seq: 46737, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9118> 2014-11-08 18:14:31.452855 7fc5ac30e700 5 -- op tracker -- seq: 46737, time: 2014-11-08 18:14:31.452855, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9117> 2014-11-08 18:14:31.452894 7fc5a56ef700 5 -- op tracker -- seq: 46737, time: 2014-11-08 18:14:31.452894, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9116> 2014-11-08 18:14:31.453453 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(a969e343/1000002baa9.00000005/head//4, version: 3473'14267, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a969e343/1000002baa9.00000005/head//4@3473'14267, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9e00 con 0x1c5c0580 -9115> 2014-11-08 18:14:31.456078 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) v1 -- ?+0 0x21839400 con 0x42c86e0 -9114> 2014-11-08 18:14:31.456118 7fc5adb11700 5 -- op tracker -- seq: 46730, time: 2014-11-08 18:14:31.456118, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4389, to_read={885c1d83/1000002b519.00000021/head//4=0,4194304}, attrs_to_read=)) -9113> 2014-11-08 18:14:31.456231 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21839400 con 0x42c86e0 -9112> 2014-11-08 18:14:31.456246 7fc5b9328700 5 -- op tracker -- seq: 46738, time: 2014-11-08 18:14:31.456108, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) -9111> 2014-11-08 18:14:31.456250 7fc5b9328700 5 -- op tracker -- seq: 46738, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) -9110> 2014-11-08 18:14:31.456252 7fc5b9328700 5 -- op tracker -- seq: 46738, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) -9109> 2014-11-08 18:14:31.456254 7fc5b9328700 5 -- op tracker -- seq: 46738, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) -9108> 2014-11-08 18:14:31.456355 7fc5b0316700 5 -- op tracker -- seq: 46738, time: 2014-11-08 18:14:31.456355, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) -9107> 2014-11-08 18:14:31.456391 7fc5b0316700 5 -- op tracker -- seq: 46738, time: 2014-11-08 18:14:31.456391, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=0)) -9106> 2014-11-08 18:14:31.458696 7fc5ad310700 5 -- op tracker -- seq: 46734, time: 2014-11-08 18:14:31.458695, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4385, attrs_read=1)) -9105> 2014-11-08 18:14:31.464327 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=fda9d484/1000001a8ae.000000c9/head//4,889bd484/1000002b2cc.00000012/head//4,2dabd484/1000002bb6e.00000006/head//4)) v1 -- ?+0 0x22ac9f80 con 0x1c5c0580 -9104> 2014-11-08 18:14:31.464364 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x22acba80 con 0x42c86e0 -9103> 2014-11-08 18:14:31.464380 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 78ebd484/1000001a34e.00000036/head//4 -9102> 2014-11-08 18:14:31.464406 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb 5fe7d484/1000002af8c.00000151/head//4) v3 -- ?+0 0x22766300 con 0x1e50aec0 -9101> 2014-11-08 18:14:31.464442 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x22acba80 con 0x42c86e0 -9100> 2014-11-08 18:14:31.464458 7fc5b9328700 5 -- op tracker -- seq: 46739, time: 2014-11-08 18:14:31.464374, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) -9099> 2014-11-08 18:14:31.464469 7fc5b9328700 5 -- op tracker -- seq: 46739, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) -9098> 2014-11-08 18:14:31.464475 7fc5b9328700 5 -- op tracker -- seq: 46739, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) -9097> 2014-11-08 18:14:31.464481 7fc5b9328700 5 -- op tracker -- seq: 46739, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) -9096> 2014-11-08 18:14:31.464520 7fc5ad310700 5 -- op tracker -- seq: 46739, time: 2014-11-08 18:14:31.464520, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) -9095> 2014-11-08 18:14:31.470975 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8449 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) v1 ==== 2097696+0+0 (1481858322 0 0) 0x1c899e00 con 0x1c5c0580 -9094> 2014-11-08 18:14:31.471007 7fc5a152e700 5 -- op tracker -- seq: 46740, time: 2014-11-08 18:14:31.400809, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) -9093> 2014-11-08 18:14:31.471015 7fc5a152e700 5 -- op tracker -- seq: 46740, time: 2014-11-08 18:14:31.400813, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) -9092> 2014-11-08 18:14:31.471018 7fc5a152e700 5 -- op tracker -- seq: 46740, time: 2014-11-08 18:14:31.470966, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) -9091> 2014-11-08 18:14:31.471020 7fc5a152e700 5 -- op tracker -- seq: 46740, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) -9090> 2014-11-08 18:14:31.471510 7fc5adb11700 5 -- op tracker -- seq: 46740, time: 2014-11-08 18:14:31.471510, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) -9089> 2014-11-08 18:14:31.475878 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(885c1d83/1000002b519.00000021/head//4, version: 3473'6275, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(885c1d83/1000002b519.00000021/head//4@3473'6275, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393dc00 con 0x1e50aec0 -9088> 2014-11-08 18:14:31.476007 7fc5adb11700 5 -- op tracker -- seq: 46740, time: 2014-11-08 18:14:31.476007, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4389, attrs_read=1)) -9087> 2014-11-08 18:14:31.540377 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8450 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x22450f00 con 0x1c5c0580 -9086> 2014-11-08 18:14:31.540411 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540283, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -9085> 2014-11-08 18:14:31.540420 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540285, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -9084> 2014-11-08 18:14:31.540425 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540349, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -9083> 2014-11-08 18:14:31.540429 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540405, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -9082> 2014-11-08 18:14:31.540434 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540434, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -9081> 2014-11-08 18:14:31.540442 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540442, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -9080> 2014-11-08 18:14:31.540449 7fc5b9b29700 5 -- op tracker -- seq: 46741, time: 2014-11-08 18:14:31.540449, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -9079> 2014-11-08 18:14:31.540491 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8451 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) v2 ==== 110+0+0 (886445270 0 0) 0x229b9e00 con 0x1c5c0580 -9078> 2014-11-08 18:14:31.540512 7fc5a152e700 5 -- op tracker -- seq: 46742, time: 2014-11-08 18:14:31.540367, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9077> 2014-11-08 18:14:31.540517 7fc5a152e700 5 -- op tracker -- seq: 46742, time: 2014-11-08 18:14:31.540367, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9076> 2014-11-08 18:14:31.540522 7fc5a152e700 5 -- op tracker -- seq: 46742, time: 2014-11-08 18:14:31.540488, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9075> 2014-11-08 18:14:31.540527 7fc5a152e700 5 -- op tracker -- seq: 46742, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9074> 2014-11-08 18:14:31.540559 7fc5ac30e700 5 -- op tracker -- seq: 46742, time: 2014-11-08 18:14:31.540559, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9073> 2014-11-08 18:14:31.540609 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8453 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) v2 ==== 110+0+0 (1823575626 0 0) 0x2393b400 con 0x1c5c0580 -9072> 2014-11-08 18:14:31.540632 7fc5ac30e700 5 -- op tracker -- seq: 46742, time: 2014-11-08 18:14:31.540632, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(9356dd13/1000001aa1e.00000034/head//4)]) -9071> 2014-11-08 18:14:31.540617 7fc5a152e700 5 -- op tracker -- seq: 46743, time: 2014-11-08 18:14:31.540581, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) -9070> 2014-11-08 18:14:31.540651 7fc5a152e700 5 -- op tracker -- seq: 46743, time: 2014-11-08 18:14:31.540581, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) -9069> 2014-11-08 18:14:31.540654 7fc5a152e700 5 -- op tracker -- seq: 46743, time: 2014-11-08 18:14:31.540608, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) -9068> 2014-11-08 18:14:31.540657 7fc5a152e700 5 -- op tracker -- seq: 46743, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) -9067> 2014-11-08 18:14:31.540673 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8452 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x224514a0 con 0x1c5c0580 -9066> 2014-11-08 18:14:31.540679 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540553, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -9065> 2014-11-08 18:14:31.540683 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540553, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -9064> 2014-11-08 18:14:31.540699 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540572, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -9063> 2014-11-08 18:14:31.540701 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540678, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -9062> 2014-11-08 18:14:31.540704 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540704, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -9061> 2014-11-08 18:14:31.540708 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540708, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -9060> 2014-11-08 18:14:31.540712 7fc5b9b29700 5 -- op tracker -- seq: 46744, time: 2014-11-08 18:14:31.540712, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -9059> 2014-11-08 18:14:31.555248 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1499 ==== pg_stats_ack(3 pgs tid 1324) v1 ==== 91+0+0 (2630001597 0 0) 0x1f4c7800 con 0x44ed6a0 -9058> 2014-11-08 18:14:31.563440 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=b1b7dd13/10000019aef.00000427/head//4)) v1 -- ?+0 0x22acde80 con 0x1e50aec0 -9057> 2014-11-08 18:14:31.563474 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x22ac9b00 con 0x42c86e0 -9056> 2014-11-08 18:14:31.563486 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=1 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is a658dd13/1000001a207.000005b8/head//4 -9055> 2014-11-08 18:14:31.563516 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb 9356dd13/1000001aa1e.00000034/head//4) v3 -- ?+0 0x22760900 con 0x1c5c0580 -9054> 2014-11-08 18:14:31.563530 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb 9356dd13/1000001aa1e.00000034/head//4) v3 -- ?+0 0x22763180 con 0x1e50aec0 -9053> 2014-11-08 18:14:31.563556 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x22ac9b00 con 0x42c86e0 -9052> 2014-11-08 18:14:31.563570 7fc5b9328700 5 -- op tracker -- seq: 46745, time: 2014-11-08 18:14:31.563480, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) -9051> 2014-11-08 18:14:31.563581 7fc5b9328700 5 -- op tracker -- seq: 46745, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) -9050> 2014-11-08 18:14:31.563586 7fc5b9328700 5 -- op tracker -- seq: 46745, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) -9049> 2014-11-08 18:14:31.563591 7fc5b9328700 5 -- op tracker -- seq: 46745, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) -9048> 2014-11-08 18:14:31.563625 7fc5ac30e700 5 -- op tracker -- seq: 46745, time: 2014-11-08 18:14:31.563625, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) -9047> 2014-11-08 18:14:31.662326 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8454 ==== MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) v1 ==== 4195327+0+0 (870262074 0 0) 0x224bdf00 con 0x1c5c0580 -9046> 2014-11-08 18:14:31.662361 7fc5a152e700 5 -- op tracker -- seq: 46746, time: 2014-11-08 18:14:31.609196, event: header_read, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) -9045> 2014-11-08 18:14:31.662370 7fc5a152e700 5 -- op tracker -- seq: 46746, time: 2014-11-08 18:14:31.609205, event: throttled, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) -9044> 2014-11-08 18:14:31.662373 7fc5a152e700 5 -- op tracker -- seq: 46746, time: 2014-11-08 18:14:31.662316, event: all_read, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) -9043> 2014-11-08 18:14:31.662377 7fc5a152e700 5 -- op tracker -- seq: 46746, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) -9042> 2014-11-08 18:14:31.662414 7fc5b0316700 5 -- op tracker -- seq: 46746, time: 2014-11-08 18:14:31.662414, event: reached_pg, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) -9041> 2014-11-08 18:14:31.670437 7fc5c362f700 5 osd.9 5773 tick -9040> 2014-11-08 18:14:31.671298 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.1as1 5773 [PushOp(891b231a/100000199b7.00000367/head//4, version: 397'982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(891b231a/100000199b7.00000367/head//4@397'982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9ab231a/1000002a767.00000093/head//4, version: 3458'5461, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9ab231a/1000002a767.00000093/head//4@3458'5461, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1ebb5c00 con 0x1e50aec0 -9039> 2014-11-08 18:14:31.689809 7fc5b0316700 5 -- op tracker -- seq: 46746, time: 2014-11-08 18:14:31.689808, event: done, op: MOSDECSubOpReadReply(4.1as0 5773 ECSubReadReply(tid=4388, attrs_read=2)) -9038> 2014-11-08 18:14:31.712187 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) v1 -- ?+0 0x1c898000 con 0x42c86e0 -9037> 2014-11-08 18:14:31.712231 7fc5ac30e700 5 -- op tracker -- seq: 46745, time: 2014-11-08 18:14:31.712231, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4391, to_read={b1b7dd13/10000019aef.00000427/head//4=0,4194304}, attrs_to_read=)) -9036> 2014-11-08 18:14:31.712233 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1c898000 con 0x42c86e0 -9035> 2014-11-08 18:14:31.712252 7fc5b9328700 5 -- op tracker -- seq: 46747, time: 2014-11-08 18:14:31.712219, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) -9034> 2014-11-08 18:14:31.712256 7fc5b9328700 5 -- op tracker -- seq: 46747, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) -9033> 2014-11-08 18:14:31.712258 7fc5b9328700 5 -- op tracker -- seq: 46747, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) -9032> 2014-11-08 18:14:31.712261 7fc5b9328700 5 -- op tracker -- seq: 46747, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) -9031> 2014-11-08 18:14:31.712291 7fc5ac30e700 5 -- op tracker -- seq: 46747, time: 2014-11-08 18:14:31.712291, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) -9030> 2014-11-08 18:14:31.712345 7fc5ac30e700 5 -- op tracker -- seq: 46747, time: 2014-11-08 18:14:31.712345, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=0)) -9029> 2014-11-08 18:14:31.790719 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 312 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:31.788535) v2 ==== 47+0+0 (1580042196 0 0) 0x202690e0 con 0x1eb9c780 -9028> 2014-11-08 18:14:31.790745 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:31.788535) v2 -- ?+0 0x24885a00 con 0x1eb9c780 -9027> 2014-11-08 18:14:31.790746 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 312 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:31.788535) v2 ==== 47+0+0 (1580042196 0 0) 0x22e5a1c0 con 0x1eb9d6a0 -9026> 2014-11-08 18:14:31.790759 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:31.788535) v2 -- ?+0 0x22e5bde0 con 0x1eb9d6a0 -9025> 2014-11-08 18:14:31.826499 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) v1 -- ?+0 0x21a4d000 con 0x42c86e0 -9024> 2014-11-08 18:14:31.826578 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4d000 con 0x42c86e0 -9023> 2014-11-08 18:14:31.826580 7fc5ad310700 5 -- op tracker -- seq: 46739, time: 2014-11-08 18:14:31.826580, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4390, to_read={fda9d484/1000001a8ae.000000c9/head//4=0,4194304,889bd484/1000002b2cc.00000012/head//4=0,4194304,2dabd484/1000002bb6e.00000006/head//4=0,4194304}, attrs_to_read=)) -9022> 2014-11-08 18:14:31.826596 7fc5b9328700 5 -- op tracker -- seq: 46748, time: 2014-11-08 18:14:31.826560, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) -9021> 2014-11-08 18:14:31.826600 7fc5b9328700 5 -- op tracker -- seq: 46748, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) -9020> 2014-11-08 18:14:31.826602 7fc5b9328700 5 -- op tracker -- seq: 46748, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) -9019> 2014-11-08 18:14:31.826604 7fc5b9328700 5 -- op tracker -- seq: 46748, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) -9018> 2014-11-08 18:14:31.826618 7fc5ad310700 5 -- op tracker -- seq: 46743, time: 2014-11-08 18:14:31.826618, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) -9017> 2014-11-08 18:14:31.826733 7fc5ad310700 5 -- op tracker -- seq: 46743, time: 2014-11-08 18:14:31.826733, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(a969e343/1000002baa9.00000005/head//4)]) -9016> 2014-11-08 18:14:31.826746 7fc5ad310700 5 -- op tracker -- seq: 46748, time: 2014-11-08 18:14:31.826746, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) -9015> 2014-11-08 18:14:31.826767 7fc5ad310700 5 -- op tracker -- seq: 46748, time: 2014-11-08 18:14:31.826767, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=0)) -9014> 2014-11-08 18:14:31.827098 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=c899e343/10000025206.00000000/head//4)) v1 -- ?+0 0x22acb840 con 0x1e50aec0 -9013> 2014-11-08 18:14:31.827115 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x22acd100 con 0x42c86e0 -9012> 2014-11-08 18:14:31.827125 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 64c9e343/10000019546.0000004b/head//4 -9011> 2014-11-08 18:14:31.827151 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb a969e343/1000002baa9.00000005/head//4) v3 -- ?+0 0x22763f00 con 0x1c5c0580 -9010> 2014-11-08 18:14:31.827185 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x22acd100 con 0x42c86e0 -9009> 2014-11-08 18:14:31.827195 7fc5b9328700 5 -- op tracker -- seq: 46749, time: 2014-11-08 18:14:31.827121, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) -9008> 2014-11-08 18:14:31.827200 7fc5b9328700 5 -- op tracker -- seq: 46749, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) -9007> 2014-11-08 18:14:31.827203 7fc5b9328700 5 -- op tracker -- seq: 46749, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) -9006> 2014-11-08 18:14:31.827205 7fc5b9328700 5 -- op tracker -- seq: 46749, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) -9005> 2014-11-08 18:14:31.827238 7fc5ad310700 5 -- op tracker -- seq: 46749, time: 2014-11-08 18:14:31.827237, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) -9004> 2014-11-08 18:14:31.904765 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) v1 -- ?+0 0x21a4d000 con 0x42c86e0 -9003> 2014-11-08 18:14:31.904816 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4d000 con 0x42c86e0 -9002> 2014-11-08 18:14:31.904830 7fc5b9328700 5 -- op tracker -- seq: 46750, time: 2014-11-08 18:14:31.904806, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) -9001> 2014-11-08 18:14:31.904838 7fc5b9328700 5 -- op tracker -- seq: 46750, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) -9000> 2014-11-08 18:14:31.904840 7fc5b9328700 5 -- op tracker -- seq: 46750, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) -8999> 2014-11-08 18:14:31.904842 7fc5b9328700 5 -- op tracker -- seq: 46750, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) -8998> 2014-11-08 18:14:31.904873 7fc5ad310700 5 -- op tracker -- seq: 46749, time: 2014-11-08 18:14:31.904873, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4392, to_read={c899e343/10000025206.00000000/head//4=0,4194304}, attrs_to_read=)) -8997> 2014-11-08 18:14:31.904903 7fc5ad310700 5 -- op tracker -- seq: 46750, time: 2014-11-08 18:14:31.904903, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) -8996> 2014-11-08 18:14:31.904941 7fc5ad310700 5 -- op tracker -- seq: 46750, time: 2014-11-08 18:14:31.904941, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=0)) -8995> 2014-11-08 18:14:31.994296 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8455 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) v1 ==== 6292958+0+0 (851099005 0 0) 0x224baf80 con 0x1c5c0580 -8994> 2014-11-08 18:14:31.994332 7fc5a152e700 5 -- op tracker -- seq: 46751, time: 2014-11-08 18:14:31.900417, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) -8993> 2014-11-08 18:14:31.994342 7fc5a152e700 5 -- op tracker -- seq: 46751, time: 2014-11-08 18:14:31.900420, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) -8992> 2014-11-08 18:14:31.994345 7fc5a152e700 5 -- op tracker -- seq: 46751, time: 2014-11-08 18:14:31.994286, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) -8991> 2014-11-08 18:14:31.994349 7fc5a152e700 5 -- op tracker -- seq: 46751, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) -8990> 2014-11-08 18:14:31.994427 7fc5ad310700 5 -- op tracker -- seq: 46751, time: 2014-11-08 18:14:31.994427, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) -8989> 2014-11-08 18:14:31.998895 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3668 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:31.994598) v2 ==== 47+0+0 (678461531 0 0) 0x23140960 con 0x1cce3440 -8988> 2014-11-08 18:14:31.998914 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:31.994598) v2 -- ?+0 0x202690e0 con 0x1cce3440 -8987> 2014-11-08 18:14:31.998809 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3668 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:31.994598) v2 ==== 47+0+0 (678461531 0 0) 0x224a30c0 con 0x1cce2ec0 -8986> 2014-11-08 18:14:31.998987 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:31.994598) v2 -- ?+0 0x22e5a1c0 con 0x1cce2ec0 -8985> 2014-11-08 18:14:32.008667 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(fda9d484/1000001a8ae.000000c9/head//4, version: 2829'2157, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fda9d484/1000001a8ae.000000c9/head//4@2829'2157, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(889bd484/1000002b2cc.00000012/head//4, version: 3468'6188, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(889bd484/1000002b2cc.00000012/head//4@3468'6188, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2dabd484/1000002bb6e.00000006/head//4, version: 3473'7292, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2dabd484/1000002bb6e.00000006/head//4@3473'7292, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393b400 con 0x1e50aec0 -8984> 2014-11-08 18:14:32.014706 7fc5ad310700 5 -- op tracker -- seq: 46751, time: 2014-11-08 18:14:32.014706, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4390, attrs_read=3)) -8983> 2014-11-08 18:14:32.048185 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.bbs0 5773 ECSubReadReply(tid=25179, attrs_read=0)) v1 -- ?+0 0x21a4d500 con 0x1c5c2680 -8982> 2014-11-08 18:14:32.048230 7fc5afb15700 5 -- op tracker -- seq: 46732, time: 2014-11-08 18:14:32.048229, event: done, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25179, to_read={e71a90bb/1000002a75e.0000002d/head//4=0,4194304,967a90bb/1000002a8cb.0000007f/head//4=0,4194304,cdfb90bb/1000002a771.0000000f/head//4=0,4194304,9d4c90bb/1000002af25.0000005e/head//4=0,4194304,27c90bb/1000002ae64.0000019e/head//4=0,4194304}, attrs_to_read=)) -8981> 2014-11-08 18:14:32.119120 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25178, attrs_read=0)) v1 -- ?+0 0x21a49900 con 0x1c5c2680 -8980> 2014-11-08 18:14:32.119163 7fc5aeb13700 5 -- op tracker -- seq: 46731, time: 2014-11-08 18:14:32.119163, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25178, to_read={68fa3227/1000002a8d8.00000082/head//4=0,4194304}, attrs_to_read=)) -8979> 2014-11-08 18:14:32.237360 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2158 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) v1 ==== 2097696+0+0 (247867713 0 0) 0x211d7580 con 0x1e50aec0 -8978> 2014-11-08 18:14:32.237396 7fc5a56ef700 5 -- op tracker -- seq: 46752, time: 2014-11-08 18:14:32.050964, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) -8977> 2014-11-08 18:14:32.237406 7fc5a56ef700 5 -- op tracker -- seq: 46752, time: 2014-11-08 18:14:32.050968, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) -8976> 2014-11-08 18:14:32.237409 7fc5a56ef700 5 -- op tracker -- seq: 46752, time: 2014-11-08 18:14:32.237350, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) -8975> 2014-11-08 18:14:32.237413 7fc5a56ef700 5 -- op tracker -- seq: 46752, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) -8974> 2014-11-08 18:14:32.237450 7fc5ac30e700 5 -- op tracker -- seq: 46752, time: 2014-11-08 18:14:32.237450, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) -8973> 2014-11-08 18:14:32.240086 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2159 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1dec9a40 con 0x1e50aec0 -8972> 2014-11-08 18:14:32.240125 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.239939, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8971> 2014-11-08 18:14:32.240133 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.239944, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8970> 2014-11-08 18:14:32.240136 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.240058, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8969> 2014-11-08 18:14:32.240152 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.240119, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8968> 2014-11-08 18:14:32.240157 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.240157, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8967> 2014-11-08 18:14:32.240166 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.240166, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8966> 2014-11-08 18:14:32.240174 7fc5b9b29700 5 -- op tracker -- seq: 46753, time: 2014-11-08 18:14:32.240174, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8965> 2014-11-08 18:14:32.240228 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2160 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1deca1c0 con 0x1e50aec0 -8964> 2014-11-08 18:14:32.240237 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240197, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8963> 2014-11-08 18:14:32.240240 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240198, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8962> 2014-11-08 18:14:32.240242 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240220, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8961> 2014-11-08 18:14:32.240245 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240235, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8960> 2014-11-08 18:14:32.240248 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240248, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8959> 2014-11-08 18:14:32.240251 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240251, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8958> 2014-11-08 18:14:32.240255 7fc5b9b29700 5 -- op tracker -- seq: 46754, time: 2014-11-08 18:14:32.240255, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8957> 2014-11-08 18:14:32.240295 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2161 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1dec8f00 con 0x1e50aec0 -8956> 2014-11-08 18:14:32.240303 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240266, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8955> 2014-11-08 18:14:32.240306 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240267, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8954> 2014-11-08 18:14:32.240309 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240287, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8953> 2014-11-08 18:14:32.240311 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240302, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8952> 2014-11-08 18:14:32.240314 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240314, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8951> 2014-11-08 18:14:32.240318 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240318, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8950> 2014-11-08 18:14:32.240321 7fc5b9b29700 5 -- op tracker -- seq: 46755, time: 2014-11-08 18:14:32.240321, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8949> 2014-11-08 18:14:32.240360 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2162 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1deceae0 con 0x1e50aec0 -8948> 2014-11-08 18:14:32.240368 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240332, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8947> 2014-11-08 18:14:32.240371 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240333, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8946> 2014-11-08 18:14:32.240374 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240352, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8945> 2014-11-08 18:14:32.240377 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240367, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8944> 2014-11-08 18:14:32.240379 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240379, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8943> 2014-11-08 18:14:32.240383 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240383, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8942> 2014-11-08 18:14:32.240387 7fc5b9b29700 5 -- op tracker -- seq: 46756, time: 2014-11-08 18:14:32.240387, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8941> 2014-11-08 18:14:32.240432 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2163 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1dece360 con 0x1e50aec0 -8940> 2014-11-08 18:14:32.240440 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240402, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8939> 2014-11-08 18:14:32.240443 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240403, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8938> 2014-11-08 18:14:32.240446 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240424, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8937> 2014-11-08 18:14:32.240449 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240439, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8936> 2014-11-08 18:14:32.240452 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240452, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8935> 2014-11-08 18:14:32.240456 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240456, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8934> 2014-11-08 18:14:32.240459 7fc5b9b29700 5 -- op tracker -- seq: 46757, time: 2014-11-08 18:14:32.240459, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8933> 2014-11-08 18:14:32.240498 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2164 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x1dec9fe0 con 0x1e50aec0 -8932> 2014-11-08 18:14:32.240506 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240470, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -8931> 2014-11-08 18:14:32.240510 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240470, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -8930> 2014-11-08 18:14:32.240513 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240491, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -8929> 2014-11-08 18:14:32.240515 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240505, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -8928> 2014-11-08 18:14:32.240518 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240518, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -8927> 2014-11-08 18:14:32.240522 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240522, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -8926> 2014-11-08 18:14:32.240525 7fc5b9b29700 5 -- op tracker -- seq: 46758, time: 2014-11-08 18:14:32.240525, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -8925> 2014-11-08 18:14:32.240566 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2165 ==== pg_trim(4.1a to 5734'8347 e5773) v1 ==== 34+0+0 (2539646313 0 0) 0x1decc1a0 con 0x1e50aec0 -8924> 2014-11-08 18:14:32.240573 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240536, event: header_read, op: pg_trim(4.1a to 5734'8347 e5773) -8923> 2014-11-08 18:14:32.240577 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240537, event: throttled, op: pg_trim(4.1a to 5734'8347 e5773) -8922> 2014-11-08 18:14:32.240579 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240557, event: all_read, op: pg_trim(4.1a to 5734'8347 e5773) -8921> 2014-11-08 18:14:32.240581 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240572, event: dispatched, op: pg_trim(4.1a to 5734'8347 e5773) -8920> 2014-11-08 18:14:32.240585 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240585, event: waiting_for_osdmap, op: pg_trim(4.1a to 5734'8347 e5773) -8919> 2014-11-08 18:14:32.240588 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240588, event: started, op: pg_trim(4.1a to 5734'8347 e5773) -8918> 2014-11-08 18:14:32.240591 7fc5b9b29700 5 -- op tracker -- seq: 46759, time: 2014-11-08 18:14:32.240591, event: done, op: pg_trim(4.1a to 5734'8347 e5773) -8917> 2014-11-08 18:14:32.240630 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2166 ==== pg_trim(4.1a to 5734'8347 e5773) v1 ==== 34+0+0 (2539646313 0 0) 0x22d57440 con 0x1e50aec0 -8916> 2014-11-08 18:14:32.240755 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240602, event: header_read, op: pg_trim(4.1a to 5734'8347 e5773) -8915> 2014-11-08 18:14:32.240762 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240602, event: throttled, op: pg_trim(4.1a to 5734'8347 e5773) -8914> 2014-11-08 18:14:32.240767 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240622, event: all_read, op: pg_trim(4.1a to 5734'8347 e5773) -8913> 2014-11-08 18:14:32.240771 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240637, event: dispatched, op: pg_trim(4.1a to 5734'8347 e5773) -8912> 2014-11-08 18:14:32.240776 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240776, event: waiting_for_osdmap, op: pg_trim(4.1a to 5734'8347 e5773) -8911> 2014-11-08 18:14:32.240782 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240782, event: started, op: pg_trim(4.1a to 5734'8347 e5773) -8910> 2014-11-08 18:14:32.240787 7fc5b9b29700 5 -- op tracker -- seq: 46760, time: 2014-11-08 18:14:32.240787, event: done, op: pg_trim(4.1a to 5734'8347 e5773) -8909> 2014-11-08 18:14:32.243723 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(b1b7dd13/10000019aef.00000427/head//4, version: 397'1019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b1b7dd13/10000019aef.00000427/head//4@397'1019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9e00 con 0x1c5c0580 -8908> 2014-11-08 18:14:32.243776 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(b1b7dd13/10000019aef.00000427/head//4, version: 397'1019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b1b7dd13/10000019aef.00000427/head//4@397'1019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1c6a7c00 con 0x1e50aec0 -8907> 2014-11-08 18:14:32.253487 7fc5ac30e700 5 -- op tracker -- seq: 46752, time: 2014-11-08 18:14:32.253487, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4391, attrs_read=1)) -8906> 2014-11-08 18:14:32.402397 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8457 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) v2 ==== 110+0+0 (3247497568 0 0) 0x229b9e00 con 0x1c5c0580 -8905> 2014-11-08 18:14:32.402447 7fc5a152e700 5 -- op tracker -- seq: 46761, time: 2014-11-08 18:14:32.402364, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8904> 2014-11-08 18:14:32.402457 7fc5a152e700 5 -- op tracker -- seq: 46761, time: 2014-11-08 18:14:32.402364, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8903> 2014-11-08 18:14:32.402463 7fc5a152e700 5 -- op tracker -- seq: 46761, time: 2014-11-08 18:14:32.402394, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8902> 2014-11-08 18:14:32.402466 7fc5a152e700 5 -- op tracker -- seq: 46761, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8901> 2014-11-08 18:14:32.402500 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8456 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x22450000 con 0x1c5c0580 -8900> 2014-11-08 18:14:32.402505 7fc5aeb13700 5 -- op tracker -- seq: 46761, time: 2014-11-08 18:14:32.402505, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8899> 2014-11-08 18:14:32.402513 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402277, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -8898> 2014-11-08 18:14:32.402516 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402281, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -8897> 2014-11-08 18:14:32.402519 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402346, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -8896> 2014-11-08 18:14:32.402522 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402509, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -8895> 2014-11-08 18:14:32.402525 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402525, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -8894> 2014-11-08 18:14:32.402532 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402532, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -8893> 2014-11-08 18:14:32.402537 7fc5b9b29700 5 -- op tracker -- seq: 46762, time: 2014-11-08 18:14:32.402536, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -8892> 2014-11-08 18:14:32.402546 7fc5aeb13700 5 -- op tracker -- seq: 46761, time: 2014-11-08 18:14:32.402546, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8891> 2014-11-08 18:14:32.435345 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2167 ==== MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 4196083+0+0 (4223371256 0 0) 0x2393b400 con 0x1e50aec0 -8890> 2014-11-08 18:14:32.435391 7fc5a56ef700 5 -- op tracker -- seq: 46763, time: 2014-11-08 18:14:32.240802, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8889> 2014-11-08 18:14:32.435405 7fc5a56ef700 5 -- op tracker -- seq: 46763, time: 2014-11-08 18:14:32.240803, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8888> 2014-11-08 18:14:32.435417 7fc5a56ef700 5 -- op tracker -- seq: 46763, time: 2014-11-08 18:14:32.435334, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8887> 2014-11-08 18:14:32.435425 7fc5a56ef700 5 -- op tracker -- seq: 46763, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8886> 2014-11-08 18:14:32.435471 7fc5acb0f700 5 -- op tracker -- seq: 46763, time: 2014-11-08 18:14:32.435471, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8885> 2014-11-08 18:14:32.441269 7fc5a56ef700 5 -- op tracker -- seq: 46763, time: 2014-11-08 18:14:32.441268, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(3f0a90b7/1000002b8d2.00000085/head//4, version: 3473'6980, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f0a90b7/1000002b8d2.00000085/head//4@3473'6980, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(435a90b7/1000002b372.0000003a/head//4, version: 3468'6261, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(435a90b7/1000002b372.0000003a/head//4@3468'6261, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8884> 2014-11-08 18:14:32.443118 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2168 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) v1 ==== 31504+0+0 (3389739391 0 0) 0x211d0f00 con 0x1e50aec0 -8883> 2014-11-08 18:14:32.443153 7fc5a56ef700 5 -- op tracker -- seq: 46764, time: 2014-11-08 18:14:32.442963, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) -8882> 2014-11-08 18:14:32.443159 7fc5a56ef700 5 -- op tracker -- seq: 46764, time: 2014-11-08 18:14:32.442967, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) -8881> 2014-11-08 18:14:32.443162 7fc5a56ef700 5 -- op tracker -- seq: 46764, time: 2014-11-08 18:14:32.443115, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) -8880> 2014-11-08 18:14:32.443164 7fc5a56ef700 5 -- op tracker -- seq: 46764, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) -8879> 2014-11-08 18:14:32.443204 7fc5ad310700 5 -- op tracker -- seq: 46764, time: 2014-11-08 18:14:32.443204, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) -8878> 2014-11-08 18:14:32.443330 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(c899e343/10000025206.00000000/head//4, version: 3033'6981, data_included: [0~30720], data_size: 30720, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c899e343/10000025206.00000000/head//4@3033'6981, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:61440, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9200 con 0x1c5c0580 -8877> 2014-11-08 18:14:32.443534 7fc5a56ef700 5 -- op tracker -- seq: 46764, time: 2014-11-08 18:14:32.443533, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4392, attrs_read=1)) -8876> 2014-11-08 18:14:32.451626 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8459 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) v2 ==== 110+0+0 (2608617421 0 0) 0x229b9200 con 0x1c5c0580 -8875> 2014-11-08 18:14:32.451659 7fc5a152e700 5 -- op tracker -- seq: 46765, time: 2014-11-08 18:14:32.451595, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) -8874> 2014-11-08 18:14:32.451667 7fc5a152e700 5 -- op tracker -- seq: 46765, time: 2014-11-08 18:14:32.451595, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) -8873> 2014-11-08 18:14:32.451669 7fc5a152e700 5 -- op tracker -- seq: 46765, time: 2014-11-08 18:14:32.451625, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) -8872> 2014-11-08 18:14:32.451672 7fc5a152e700 5 -- op tracker -- seq: 46765, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) -8871> 2014-11-08 18:14:32.451750 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8458 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x22455dc0 con 0x1c5c0580 -8870> 2014-11-08 18:14:32.451761 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451512, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8869> 2014-11-08 18:14:32.451764 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451515, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8868> 2014-11-08 18:14:32.451766 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451579, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8867> 2014-11-08 18:14:32.451769 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451758, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8866> 2014-11-08 18:14:32.451772 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451772, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8865> 2014-11-08 18:14:32.451778 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451777, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8864> 2014-11-08 18:14:32.451782 7fc5b9b29700 5 -- op tracker -- seq: 46766, time: 2014-11-08 18:14:32.451782, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8863> 2014-11-08 18:14:32.451805 7fc5afb15700 5 -- op tracker -- seq: 46765, time: 2014-11-08 18:14:32.451805, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) -8862> 2014-11-08 18:14:32.451868 7fc5afb15700 5 -- op tracker -- seq: 46765, time: 2014-11-08 18:14:32.451868, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(c899e343/10000025206.00000000/head//4)]) -8861> 2014-11-08 18:14:32.452155 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=64c9e343/10000019546.0000004b/head//4)) v1 -- ?+0 0x22acd580 con 0x1e50aec0 -8860> 2014-11-08 18:14:32.452172 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x22aca1c0 con 0x42c86e0 -8859> 2014-11-08 18:14:32.452240 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x22aca1c0 con 0x42c86e0 -8858> 2014-11-08 18:14:32.452250 7fc5b9328700 5 -- op tracker -- seq: 46767, time: 2014-11-08 18:14:32.452178, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) -8857> 2014-11-08 18:14:32.452254 7fc5b9328700 5 -- op tracker -- seq: 46767, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) -8856> 2014-11-08 18:14:32.452257 7fc5b9328700 5 -- op tracker -- seq: 46767, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) -8855> 2014-11-08 18:14:32.452259 7fc5b9328700 5 -- op tracker -- seq: 46767, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) -8854> 2014-11-08 18:14:32.452182 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 855de343/1000002a838.0000004b/head//4 -8853> 2014-11-08 18:14:32.452288 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb c899e343/10000025206.00000000/head//4) v3 -- ?+0 0x22760000 con 0x1c5c0580 -8852> 2014-11-08 18:14:32.452305 7fc5ad310700 5 -- op tracker -- seq: 46767, time: 2014-11-08 18:14:32.452305, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) -8851> 2014-11-08 18:14:32.459131 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2169 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x22d57620 con 0x1e50aec0 -8850> 2014-11-08 18:14:32.459153 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459069, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -8849> 2014-11-08 18:14:32.459158 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459071, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -8848> 2014-11-08 18:14:32.459161 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459113, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -8847> 2014-11-08 18:14:32.459163 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459150, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -8846> 2014-11-08 18:14:32.459166 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459166, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -8845> 2014-11-08 18:14:32.459173 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459173, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -8844> 2014-11-08 18:14:32.459177 7fc5b9b29700 5 -- op tracker -- seq: 46768, time: 2014-11-08 18:14:32.459177, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -8843> 2014-11-08 18:14:32.459229 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2170 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x22d55280 con 0x1e50aec0 -8842> 2014-11-08 18:14:32.459239 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459198, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -8841> 2014-11-08 18:14:32.459242 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459199, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -8840> 2014-11-08 18:14:32.459245 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459220, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -8839> 2014-11-08 18:14:32.459247 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459238, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -8838> 2014-11-08 18:14:32.459249 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459249, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -8837> 2014-11-08 18:14:32.459253 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459253, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -8836> 2014-11-08 18:14:32.459257 7fc5b9b29700 5 -- op tracker -- seq: 46769, time: 2014-11-08 18:14:32.459257, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -8835> 2014-11-08 18:14:32.459304 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2171 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x22d54b00 con 0x1e50aec0 -8834> 2014-11-08 18:14:32.459313 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459273, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -8833> 2014-11-08 18:14:32.459316 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459274, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -8832> 2014-11-08 18:14:32.459319 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459295, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -8831> 2014-11-08 18:14:32.459321 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459312, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -8830> 2014-11-08 18:14:32.459324 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459324, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -8829> 2014-11-08 18:14:32.459327 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459327, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -8828> 2014-11-08 18:14:32.459331 7fc5b9b29700 5 -- op tracker -- seq: 46770, time: 2014-11-08 18:14:32.459331, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -8827> 2014-11-08 18:14:32.459382 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2172 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x22d51e00 con 0x1e50aec0 -8826> 2014-11-08 18:14:32.459391 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459347, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -8825> 2014-11-08 18:14:32.459394 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459347, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -8824> 2014-11-08 18:14:32.459397 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459373, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -8823> 2014-11-08 18:14:32.459399 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459390, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -8822> 2014-11-08 18:14:32.459402 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459402, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -8821> 2014-11-08 18:14:32.459405 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459405, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -8820> 2014-11-08 18:14:32.459408 7fc5b9b29700 5 -- op tracker -- seq: 46771, time: 2014-11-08 18:14:32.459408, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -8819> 2014-11-08 18:14:32.496398 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) v1 -- ?+0 0x224baf80 con 0x42c86e0 -8818> 2014-11-08 18:14:32.496435 7fc5ad310700 5 -- op tracker -- seq: 46767, time: 2014-11-08 18:14:32.496434, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4393, to_read={64c9e343/10000019546.0000004b/head//4=0,4194304}, attrs_to_read=)) -8817> 2014-11-08 18:14:32.496467 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224baf80 con 0x42c86e0 -8816> 2014-11-08 18:14:32.496477 7fc5b9328700 5 -- op tracker -- seq: 46772, time: 2014-11-08 18:14:32.496426, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) -8815> 2014-11-08 18:14:32.496480 7fc5b9328700 5 -- op tracker -- seq: 46772, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) -8814> 2014-11-08 18:14:32.496482 7fc5b9328700 5 -- op tracker -- seq: 46772, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) -8813> 2014-11-08 18:14:32.496484 7fc5b9328700 5 -- op tracker -- seq: 46772, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) -8812> 2014-11-08 18:14:32.496515 7fc5afb15700 5 -- op tracker -- seq: 46772, time: 2014-11-08 18:14:32.496515, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) -8811> 2014-11-08 18:14:32.496547 7fc5afb15700 5 -- op tracker -- seq: 46772, time: 2014-11-08 18:14:32.496547, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=0)) -8810> 2014-11-08 18:14:32.532600 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1d1b6ae0 con 0x1e50aec0 -8809> 2014-11-08 18:14:32.532634 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1d1b1c20 con 0x1e50aec0 -8808> 2014-11-08 18:14:32.532644 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(3f0a90b7/1000002b8d2.00000085/head//4),PushReplyOp(435a90b7/1000002b372.0000003a/head//4)]) v2 -- ?+0 0x1d5b1000 con 0x1e50aec0 -8807> 2014-11-08 18:14:32.670540 7fc5c362f700 5 osd.9 5773 tick -8806> 2014-11-08 18:14:32.704529 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2173 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) v1 ==== 2097696+0+0 (2143568169 0 0) 0x211d0f00 con 0x1e50aec0 -8805> 2014-11-08 18:14:32.704567 7fc5a56ef700 5 -- op tracker -- seq: 46773, time: 2014-11-08 18:14:32.609790, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) -8804> 2014-11-08 18:14:32.704581 7fc5a56ef700 5 -- op tracker -- seq: 46773, time: 2014-11-08 18:14:32.609793, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) -8803> 2014-11-08 18:14:32.704590 7fc5a56ef700 5 -- op tracker -- seq: 46773, time: 2014-11-08 18:14:32.704503, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) -8802> 2014-11-08 18:14:32.704599 7fc5a56ef700 5 -- op tracker -- seq: 46773, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) -8801> 2014-11-08 18:14:32.704640 7fc5ad310700 5 -- op tracker -- seq: 46773, time: 2014-11-08 18:14:32.704639, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) -8800> 2014-11-08 18:14:32.708394 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(64c9e343/10000019546.0000004b/head//4, version: 397'1327, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(64c9e343/10000019546.0000004b/head//4@397'1327, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229be600 con 0x1c5c0580 -8799> 2014-11-08 18:14:32.712357 7fc5a56ef700 5 -- op tracker -- seq: 46773, time: 2014-11-08 18:14:32.712357, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4393, attrs_read=1)) -8798> 2014-11-08 18:14:32.838461 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8461 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) v2 ==== 110+0+0 (1145300413 0 0) 0x229be600 con 0x1c5c0580 -8797> 2014-11-08 18:14:32.838499 7fc5a152e700 5 -- op tracker -- seq: 46774, time: 2014-11-08 18:14:32.838411, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) -8796> 2014-11-08 18:14:32.838508 7fc5a152e700 5 -- op tracker -- seq: 46774, time: 2014-11-08 18:14:32.838411, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) -8795> 2014-11-08 18:14:32.838512 7fc5a152e700 5 -- op tracker -- seq: 46774, time: 2014-11-08 18:14:32.838460, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) -8794> 2014-11-08 18:14:32.838515 7fc5a152e700 5 -- op tracker -- seq: 46774, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) -8793> 2014-11-08 18:14:32.838558 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8460 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x22453de0 con 0x1c5c0580 -8792> 2014-11-08 18:14:32.838570 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838306, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8791> 2014-11-08 18:14:32.838574 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838311, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8790> 2014-11-08 18:14:32.838576 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838389, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8789> 2014-11-08 18:14:32.838579 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838566, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8788> 2014-11-08 18:14:32.838582 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838582, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8787> 2014-11-08 18:14:32.838589 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838589, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8786> 2014-11-08 18:14:32.838595 7fc5b9b29700 5 -- op tracker -- seq: 46775, time: 2014-11-08 18:14:32.838595, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8785> 2014-11-08 18:14:32.838705 7fc5afb15700 5 -- op tracker -- seq: 46774, time: 2014-11-08 18:14:32.838705, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) -8784> 2014-11-08 18:14:32.838778 7fc5afb15700 5 -- op tracker -- seq: 46774, time: 2014-11-08 18:14:32.838778, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(64c9e343/10000019546.0000004b/head//4)]) -8783> 2014-11-08 18:14:32.839054 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=855de343/1000002a838.0000004b/head//4)) v1 -- ?+0 0x22ace540 con 0x1e50aec0 -8782> 2014-11-08 18:14:32.839074 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x21a533c0 con 0x42c86e0 -8781> 2014-11-08 18:14:32.839086 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 9ebde343/1000001a691.000000f5/head//4 -8780> 2014-11-08 18:14:32.839108 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb 64c9e343/10000019546.0000004b/head//4) v3 -- ?+0 0x1d33d100 con 0x1c5c0580 -8779> 2014-11-08 18:14:32.839190 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x21a533c0 con 0x42c86e0 -8778> 2014-11-08 18:14:32.839202 7fc5b9328700 5 -- op tracker -- seq: 46776, time: 2014-11-08 18:14:32.839082, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) -8777> 2014-11-08 18:14:32.839208 7fc5b9328700 5 -- op tracker -- seq: 46776, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) -8776> 2014-11-08 18:14:32.839211 7fc5b9328700 5 -- op tracker -- seq: 46776, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) -8775> 2014-11-08 18:14:32.839215 7fc5b9328700 5 -- op tracker -- seq: 46776, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) -8774> 2014-11-08 18:14:32.839246 7fc5ad310700 5 -- op tracker -- seq: 46776, time: 2014-11-08 18:14:32.839246, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) -8773> 2014-11-08 18:14:32.951177 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) v1 -- ?+0 0x21a4d000 con 0x42c86e0 -8772> 2014-11-08 18:14:32.951214 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4d000 con 0x42c86e0 -8771> 2014-11-08 18:14:32.951227 7fc5b9328700 5 -- op tracker -- seq: 46777, time: 2014-11-08 18:14:32.951204, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) -8770> 2014-11-08 18:14:32.951236 7fc5b9328700 5 -- op tracker -- seq: 46777, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) -8769> 2014-11-08 18:14:32.951240 7fc5b9328700 5 -- op tracker -- seq: 46777, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) -8768> 2014-11-08 18:14:32.951242 7fc5b9328700 5 -- op tracker -- seq: 46777, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) -8767> 2014-11-08 18:14:32.951275 7fc5ad310700 5 -- op tracker -- seq: 46776, time: 2014-11-08 18:14:32.951274, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4394, to_read={855de343/1000002a838.0000004b/head//4=0,4194304}, attrs_to_read=)) -8766> 2014-11-08 18:14:32.951289 7fc5afb15700 5 -- op tracker -- seq: 46777, time: 2014-11-08 18:14:32.951289, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) -8765> 2014-11-08 18:14:32.951324 7fc5afb15700 5 -- op tracker -- seq: 46777, time: 2014-11-08 18:14:32.951324, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=0)) -8764> 2014-11-08 18:14:33.219220 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1569 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3463146709 0 0) 0x228798c0 con 0x21468c60 -8763> 2014-11-08 18:14:33.219252 7fc5a39d2700 5 -- op tracker -- seq: 46778, time: 2014-11-08 18:14:33.219112, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) -8762> 2014-11-08 18:14:33.219263 7fc5a39d2700 5 -- op tracker -- seq: 46778, time: 2014-11-08 18:14:33.219117, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) -8761> 2014-11-08 18:14:33.219268 7fc5a39d2700 5 -- op tracker -- seq: 46778, time: 2014-11-08 18:14:33.219205, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) -8760> 2014-11-08 18:14:33.219274 7fc5a39d2700 5 -- op tracker -- seq: 46778, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) -8759> 2014-11-08 18:14:33.219323 7fc5af314700 5 -- op tracker -- seq: 46778, time: 2014-11-08 18:14:33.219323, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) -8758> 2014-11-08 18:14:33.241969 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3702 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:33.239557) v2 ==== 47+0+0 (836357406 0 0) 0x2336f620 con 0x1c8b7d20 -8757> 2014-11-08 18:14:33.241993 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:33.239557) v2 -- ?+0 0x23140960 con 0x1c8b7d20 -8756> 2014-11-08 18:14:33.242059 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3702 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:33.239557) v2 ==== 47+0+0 (836357406 0 0) 0x1d041680 con 0x1c9f8580 -8755> 2014-11-08 18:14:33.242070 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:33.239557) v2 -- ?+0 0x224a30c0 con 0x1c9f8580 -8754> 2014-11-08 18:14:33.287453 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2174 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) v1 ==== 2097696+0+0 (3731317345 0 0) 0x211d0f00 con 0x1e50aec0 -8753> 2014-11-08 18:14:33.287494 7fc5a56ef700 5 -- op tracker -- seq: 46779, time: 2014-11-08 18:14:33.223967, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) -8752> 2014-11-08 18:14:33.287508 7fc5a56ef700 5 -- op tracker -- seq: 46779, time: 2014-11-08 18:14:33.223969, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) -8751> 2014-11-08 18:14:33.287518 7fc5a56ef700 5 -- op tracker -- seq: 46779, time: 2014-11-08 18:14:33.287426, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) -8750> 2014-11-08 18:14:33.287526 7fc5a56ef700 5 -- op tracker -- seq: 46779, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) -8749> 2014-11-08 18:14:33.287567 7fc5afb15700 5 -- op tracker -- seq: 46779, time: 2014-11-08 18:14:33.287567, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) -8748> 2014-11-08 18:14:33.291369 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(855de343/1000002a838.0000004b/head//4, version: 3458'10672, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(855de343/1000002a838.0000004b/head//4@3458'10672, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229be600 con 0x1c5c0580 -8747> 2014-11-08 18:14:33.294521 7fc5a56ef700 5 -- op tracker -- seq: 46779, time: 2014-11-08 18:14:33.294520, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4394, attrs_read=1)) -8746> 2014-11-08 18:14:33.312042 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59228, attrs_read=0)) v1 -- ?+0 0x1d5fc880 con 0x21468c60 -8745> 2014-11-08 18:14:33.312089 7fc5af314700 5 -- op tracker -- seq: 46778, time: 2014-11-08 18:14:33.312089, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59228, to_read={a5ad314e/1000002aa1d.000000c6/head//4=0,4194304}, attrs_to_read=)) -8744> 2014-11-08 18:14:33.453305 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1570 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) v3 ==== 757+0+0 (2357559763 0 0) 0x1e58bf00 con 0x21468c60 -8743> 2014-11-08 18:14:33.453343 7fc5a39d2700 5 -- op tracker -- seq: 46780, time: 2014-11-08 18:14:33.453227, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8742> 2014-11-08 18:14:33.453348 7fc5a39d2700 5 -- op tracker -- seq: 46780, time: 2014-11-08 18:14:33.453229, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8741> 2014-11-08 18:14:33.453351 7fc5a39d2700 5 -- op tracker -- seq: 46780, time: 2014-11-08 18:14:33.453300, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8740> 2014-11-08 18:14:33.453355 7fc5a39d2700 5 -- op tracker -- seq: 46780, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8739> 2014-11-08 18:14:33.453515 7fc5acb0f700 5 -- op tracker -- seq: 46780, time: 2014-11-08 18:14:33.453515, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8738> 2014-11-08 18:14:33.453527 7fc5acb0f700 5 -- op tracker -- seq: 46780, time: 2014-11-08 18:14:33.453526, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8737> 2014-11-08 18:14:33.453575 7fc5acb0f700 5 -- op tracker -- seq: 46780, time: 2014-11-08 18:14:33.453575, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 6f8abddd/1000002a98a.0000012d/head//4) -8736> 2014-11-08 18:14:33.552716 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8462 ==== MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dc1c5b16/ 1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10035484+0+0 (2311360363 0 0) 0x229be600 con 0x1c5c0580 -8735> 2014-11-08 18:14:33.552779 7fc5a152e700 5 -- op tracker -- seq: 46781, time: 2014-11-08 18:14:33.413860, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( dc1c5b16/1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8734> 2014-11-08 18:14:33.552809 7fc5a152e700 5 -- op tracker -- seq: 46781, time: 2014-11-08 18:14:33.413862, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dc1c5b16/ 1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8733> 2014-11-08 18:14:33.552835 7fc5a152e700 5 -- op tracker -- seq: 46781, time: 2014-11-08 18:14:33.552678, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dc1c5b16/ 1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8732> 2014-11-08 18:14:33.552861 7fc5a152e700 5 -- op tracker -- seq: 46781, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dc1c5b16/ 1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8731> 2014-11-08 18:14:33.552922 7fc5ad310700 5 -- op tracker -- seq: 46781, time: 2014-11-08 18:14:33.552922, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( dc1c5b16/1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8730> 2014-11-08 18:14:33.557846 7fc5a152e700 5 -- op tracker -- seq: 46781, time: 2014-11-08 18:14:33.557845, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(c81b5b16/1000001a81f.00000042/head//4, version: 2829'2191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c81b5b16/1000001a81f.00000042/head//4@2829'2191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e8b5b16/1000002acf9.00000012/head//4, version: 3468'5996, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e8b5b16/1000002acf9.00000012/head//4@3468'5996, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4eeb5b16/100000283e7.000000a1/head//4, version: 3033'3686, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4eeb5b16/100000283e7.000000a1/head//4@3033'3686, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dc1c5b16/1000000e4b7.00000372/head//4, version: 397'199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dc1c5b16/ 1000000e4b7.00000372/head//4@397'199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b9c5b16/100000153b6.00000001/head//4, version: 397'453, data_included: [0~1642496], data_size: 1642496, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b9c5b16/100000153b6.00000001/head//4@397'453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3284992, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8729> 2014-11-08 18:14:33.558127 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8463 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x2286fbc0 con 0x1c5c0580 -8728> 2014-11-08 18:14:33.558144 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.557986, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8727> 2014-11-08 18:14:33.558148 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.557988, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8726> 2014-11-08 18:14:33.558150 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.558056, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8725> 2014-11-08 18:14:33.558153 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.558140, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8724> 2014-11-08 18:14:33.558156 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.558156, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8723> 2014-11-08 18:14:33.558165 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.558165, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8722> 2014-11-08 18:14:33.558170 7fc5b9b29700 5 -- op tracker -- seq: 46782, time: 2014-11-08 18:14:33.558170, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8721> 2014-11-08 18:14:33.558238 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8464 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) v2 ==== 110+0+0 (1596536580 0 0) 0x229be600 con 0x1c5c0580 -8720> 2014-11-08 18:14:33.558254 7fc5a152e700 5 -- op tracker -- seq: 46783, time: 2014-11-08 18:14:33.558197, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) -8719> 2014-11-08 18:14:33.558264 7fc5a152e700 5 -- op tracker -- seq: 46783, time: 2014-11-08 18:14:33.558198, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) -8718> 2014-11-08 18:14:33.558273 7fc5a152e700 5 -- op tracker -- seq: 46783, time: 2014-11-08 18:14:33.558229, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) -8717> 2014-11-08 18:14:33.558282 7fc5a152e700 5 -- op tracker -- seq: 46783, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) -8716> 2014-11-08 18:14:33.558331 7fc5afb15700 5 -- op tracker -- seq: 46783, time: 2014-11-08 18:14:33.558331, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) -8715> 2014-11-08 18:14:33.558605 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=9ebde343/1000001a691.000000f5/head//4)) v1 -- ?+0 0x21a506c0 con 0x1e50aec0 -8714> 2014-11-08 18:14:33.558624 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x21a55c40 con 0x42c86e0 -8713> 2014-11-08 18:14:33.558634 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is f3ffe343/1000002b4e5.0000002b/head//4 -8712> 2014-11-08 18:14:33.558654 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb 855de343/1000002a838.0000004b/head//4) v3 -- ?+0 0x1d33b600 con 0x1c5c0580 -8711> 2014-11-08 18:14:33.558730 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x21a55c40 con 0x42c86e0 -8710> 2014-11-08 18:14:33.558739 7fc5b9328700 5 -- op tracker -- seq: 46784, time: 2014-11-08 18:14:33.558630, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) -8709> 2014-11-08 18:14:33.558744 7fc5b9328700 5 -- op tracker -- seq: 46784, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) -8708> 2014-11-08 18:14:33.558746 7fc5b9328700 5 -- op tracker -- seq: 46784, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) -8707> 2014-11-08 18:14:33.558749 7fc5b9328700 5 -- op tracker -- seq: 46784, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) -8706> 2014-11-08 18:14:33.558833 7fc5a152e700 5 -- op tracker -- seq: 46783, time: 2014-11-08 18:14:33.558833, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(855de343/1000002a838.0000004b/head//4)]) -8705> 2014-11-08 18:14:33.558871 7fc5ad310700 5 -- op tracker -- seq: 46784, time: 2014-11-08 18:14:33.558871, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) -8704> 2014-11-08 18:14:33.640032 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) v1 -- ?+0 0x2183b980 con 0x42c86e0 -8703> 2014-11-08 18:14:33.640083 7fc5ad310700 5 -- op tracker -- seq: 46784, time: 2014-11-08 18:14:33.640083, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4395, to_read={9ebde343/1000001a691.000000f5/head//4=0,4194304}, attrs_to_read=)) -8702> 2014-11-08 18:14:33.640124 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x2183b980 con 0x42c86e0 -8701> 2014-11-08 18:14:33.640136 7fc5b9328700 5 -- op tracker -- seq: 46785, time: 2014-11-08 18:14:33.640071, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) -8700> 2014-11-08 18:14:33.640139 7fc5b9328700 5 -- op tracker -- seq: 46785, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) -8699> 2014-11-08 18:14:33.640141 7fc5b9328700 5 -- op tracker -- seq: 46785, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) -8698> 2014-11-08 18:14:33.640143 7fc5b9328700 5 -- op tracker -- seq: 46785, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) -8697> 2014-11-08 18:14:33.640174 7fc5afb15700 5 -- op tracker -- seq: 46785, time: 2014-11-08 18:14:33.640174, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) -8696> 2014-11-08 18:14:33.640211 7fc5afb15700 5 -- op tracker -- seq: 46785, time: 2014-11-08 18:14:33.640211, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=0)) -8695> 2014-11-08 18:14:33.670652 7fc5c362f700 5 osd.9 5773 tick -8694> 2014-11-08 18:14:33.762641 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2175 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) v1 ==== 2097696+0+0 (1015030330 0 0) 0x211d0f00 con 0x1e50aec0 -8693> 2014-11-08 18:14:33.762672 7fc5a56ef700 5 -- op tracker -- seq: 46786, time: 2014-11-08 18:14:33.670155, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) -8692> 2014-11-08 18:14:33.762681 7fc5a56ef700 5 -- op tracker -- seq: 46786, time: 2014-11-08 18:14:33.670161, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) -8691> 2014-11-08 18:14:33.762693 7fc5a56ef700 5 -- op tracker -- seq: 46786, time: 2014-11-08 18:14:33.762632, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) -8690> 2014-11-08 18:14:33.762696 7fc5a56ef700 5 -- op tracker -- seq: 46786, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) -8689> 2014-11-08 18:14:33.762751 7fc5ad310700 5 -- op tracker -- seq: 46786, time: 2014-11-08 18:14:33.762750, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) -8688> 2014-11-08 18:14:33.766371 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(9ebde343/1000001a691.000000f5/head//4, version: 397'4059, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9ebde343/1000001a691.000000f5/head//4@397'4059, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229bd000 con 0x1c5c0580 -8687> 2014-11-08 18:14:33.766442 7fc5ad310700 5 -- op tracker -- seq: 46786, time: 2014-11-08 18:14:33.766442, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4395, attrs_read=1)) -8686> 2014-11-08 18:14:33.792875 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2127b840 con 0x1c5c0580 -8685> 2014-11-08 18:14:33.792906 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d55460 con 0x1c5c0580 -8684> 2014-11-08 18:14:33.792912 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d505a0 con 0x1c5c0580 -8683> 2014-11-08 18:14:33.792918 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x24959860 con 0x1c5c0580 -8682> 2014-11-08 18:14:33.792923 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495f9e0 con 0x1c5c0580 -8681> 2014-11-08 18:14:33.792931 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(c81b5b16/1000001a81f.00000042/head//4),PushReplyOp(e8b5b16/1000002acf9.00000012/head//4),PushReplyOp(4eeb5b16/100000283e7.000000a1/head//4),PushReplyOp(dc1c5b16/1000000e4b7.00000372/head//4),PushReplyOp(b9c5b16/100000153b6.00000001/head//4)]) v2 -- ?+0 0x229bc200 con 0x1c5c0580 -8680> 2014-11-08 18:14:33.804729 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8465 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) v3 ==== 757+0+0 (1978346285 0 0) 0x1d651200 con 0x1c5c0580 -8679> 2014-11-08 18:14:33.804747 7fc5a152e700 5 -- op tracker -- seq: 46787, time: 2014-11-08 18:14:33.804668, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8678> 2014-11-08 18:14:33.804756 7fc5a152e700 5 -- op tracker -- seq: 46787, time: 2014-11-08 18:14:33.804670, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8677> 2014-11-08 18:14:33.804760 7fc5a152e700 5 -- op tracker -- seq: 46787, time: 2014-11-08 18:14:33.804723, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8676> 2014-11-08 18:14:33.804781 7fc5a152e700 5 -- op tracker -- seq: 46787, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8675> 2014-11-08 18:14:33.804813 7fc5afb15700 5 -- op tracker -- seq: 46787, time: 2014-11-08 18:14:33.804813, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8674> 2014-11-08 18:14:33.804825 7fc5afb15700 5 -- op tracker -- seq: 46787, time: 2014-11-08 18:14:33.804825, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8673> 2014-11-08 18:14:33.804868 7fc5afb15700 5 -- op tracker -- seq: 46787, time: 2014-11-08 18:14:33.804868, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b9c5b16/100000153b6.00000001/head//4) -8672> 2014-11-08 18:14:33.825087 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8466 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x2495f9e0 con 0x1c5c0580 -8671> 2014-11-08 18:14:33.825107 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.824992, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8670> 2014-11-08 18:14:33.825115 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.824994, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8669> 2014-11-08 18:14:33.825120 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.825066, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8668> 2014-11-08 18:14:33.825124 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.825102, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8667> 2014-11-08 18:14:33.825129 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.825129, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8666> 2014-11-08 18:14:33.825138 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.825138, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8665> 2014-11-08 18:14:33.825128 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8467 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) v2 ==== 110+0+0 (945291628 0 0) 0x229bc200 con 0x1c5c0580 -8664> 2014-11-08 18:14:33.825145 7fc5b9b29700 5 -- op tracker -- seq: 46788, time: 2014-11-08 18:14:33.825145, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8663> 2014-11-08 18:14:33.825153 7fc5a152e700 5 -- op tracker -- seq: 46789, time: 2014-11-08 18:14:33.825087, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) -8662> 2014-11-08 18:14:33.825167 7fc5a152e700 5 -- op tracker -- seq: 46789, time: 2014-11-08 18:14:33.825088, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) -8661> 2014-11-08 18:14:33.825172 7fc5a152e700 5 -- op tracker -- seq: 46789, time: 2014-11-08 18:14:33.825124, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) -8660> 2014-11-08 18:14:33.825177 7fc5a152e700 5 -- op tracker -- seq: 46789, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) -8659> 2014-11-08 18:14:33.825205 7fc5ad310700 5 -- op tracker -- seq: 46789, time: 2014-11-08 18:14:33.825205, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) -8658> 2014-11-08 18:14:33.825269 7fc5ad310700 5 -- op tracker -- seq: 46789, time: 2014-11-08 18:14:33.825269, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(9ebde343/1000001a691.000000f5/head//4)]) -8657> 2014-11-08 18:14:33.825569 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=f3ffe343/1000002b4e5.0000002b/head//4)) v1 -- ?+0 0x21a50b40 con 0x1e50aec0 -8656> 2014-11-08 18:14:33.825585 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x21a54ec0 con 0x42c86e0 -8655> 2014-11-08 18:14:33.825596 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 5d90f343/1000002b421.0000001e/head//4 -8654> 2014-11-08 18:14:33.825638 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb 9ebde343/1000001a691.000000f5/head//4) v3 -- ?+0 0x1d33ec00 con 0x1c5c0580 -8653> 2014-11-08 18:14:33.825657 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x21a54ec0 con 0x42c86e0 -8652> 2014-11-08 18:14:33.825671 7fc5b9328700 5 -- op tracker -- seq: 46790, time: 2014-11-08 18:14:33.825591, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) -8651> 2014-11-08 18:14:33.825675 7fc5b9328700 5 -- op tracker -- seq: 46790, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) -8650> 2014-11-08 18:14:33.825678 7fc5b9328700 5 -- op tracker -- seq: 46790, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) -8649> 2014-11-08 18:14:33.825681 7fc5b9328700 5 -- op tracker -- seq: 46790, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) -8648> 2014-11-08 18:14:33.825707 7fc5afb15700 5 -- op tracker -- seq: 46790, time: 2014-11-08 18:14:33.825707, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) -8647> 2014-11-08 18:14:33.890114 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) v1 -- ?+0 0x2183b980 con 0x42c86e0 -8646> 2014-11-08 18:14:33.890148 7fc5afb15700 5 -- op tracker -- seq: 46790, time: 2014-11-08 18:14:33.890148, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4396, to_read={f3ffe343/1000002b4e5.0000002b/head//4=0,4194304}, attrs_to_read=)) -8645> 2014-11-08 18:14:33.890182 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x2183b980 con 0x42c86e0 -8644> 2014-11-08 18:14:33.890194 7fc5b9328700 5 -- op tracker -- seq: 46791, time: 2014-11-08 18:14:33.890138, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) -8643> 2014-11-08 18:14:33.890199 7fc5b9328700 5 -- op tracker -- seq: 46791, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) -8642> 2014-11-08 18:14:33.890201 7fc5b9328700 5 -- op tracker -- seq: 46791, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) -8641> 2014-11-08 18:14:33.890203 7fc5b9328700 5 -- op tracker -- seq: 46791, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) -8640> 2014-11-08 18:14:33.890232 7fc5ad310700 5 -- op tracker -- seq: 46791, time: 2014-11-08 18:14:33.890232, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) -8639> 2014-11-08 18:14:33.890269 7fc5ad310700 5 -- op tracker -- seq: 46791, time: 2014-11-08 18:14:33.890269, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=0)) -8638> 2014-11-08 18:14:34.039924 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8468 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) v3 ==== 757+0+0 (1702122757 0 0) 0x1d33ec00 con 0x1c5c0580 -8637> 2014-11-08 18:14:34.039950 7fc5a152e700 5 -- op tracker -- seq: 46792, time: 2014-11-08 18:14:34.039840, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8636> 2014-11-08 18:14:34.039958 7fc5a152e700 5 -- op tracker -- seq: 46792, time: 2014-11-08 18:14:34.039843, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8635> 2014-11-08 18:14:34.039961 7fc5a152e700 5 -- op tracker -- seq: 46792, time: 2014-11-08 18:14:34.039919, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8634> 2014-11-08 18:14:34.039966 7fc5a152e700 5 -- op tracker -- seq: 46792, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8633> 2014-11-08 18:14:34.040008 7fc5adb11700 5 -- op tracker -- seq: 46792, time: 2014-11-08 18:14:34.040008, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8632> 2014-11-08 18:14:34.040029 7fc5adb11700 5 -- op tracker -- seq: 46792, time: 2014-11-08 18:14:34.040029, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8631> 2014-11-08 18:14:34.040073 7fc5adb11700 5 -- op tracker -- seq: 46792, time: 2014-11-08 18:14:34.040073, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb ebf3fcb5/1000001a9f2.0000022f/head//4) -8630> 2014-11-08 18:14:34.072197 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2176 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) v1 ==== 2097696+0+0 (2163581352 0 0) 0x211d7d00 con 0x1e50aec0 -8629> 2014-11-08 18:14:34.072239 7fc5a56ef700 5 -- op tracker -- seq: 46793, time: 2014-11-08 18:14:34.017449, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) -8628> 2014-11-08 18:14:34.072249 7fc5a56ef700 5 -- op tracker -- seq: 46793, time: 2014-11-08 18:14:34.017451, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) -8627> 2014-11-08 18:14:34.072254 7fc5a56ef700 5 -- op tracker -- seq: 46793, time: 2014-11-08 18:14:34.072173, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) -8626> 2014-11-08 18:14:34.072257 7fc5a56ef700 5 -- op tracker -- seq: 46793, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) -8625> 2014-11-08 18:14:34.072292 7fc5afb15700 5 -- op tracker -- seq: 46793, time: 2014-11-08 18:14:34.072292, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) -8624> 2014-11-08 18:14:34.083322 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(f3ffe343/1000002b4e5.0000002b/head//4, version: 3473'12660, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f3ffe343/1000002b4e5.0000002b/head//4@3473'12660, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9200 con 0x1c5c0580 -8623> 2014-11-08 18:14:34.083539 7fc5afb15700 5 -- op tracker -- seq: 46793, time: 2014-11-08 18:14:34.083538, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4396, attrs_read=1)) -8622> 2014-11-08 18:14:34.134059 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1571 ==== MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8392121+0+0 (2457495320 0 0) 0x23938c00 con 0x21468c60 -8621> 2014-11-08 18:14:34.134122 7fc5a39d2700 5 -- op tracker -- seq: 46794, time: 2014-11-08 18:14:33.950151, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8620> 2014-11-08 18:14:34.134148 7fc5a39d2700 5 -- op tracker -- seq: 46794, time: 2014-11-08 18:14:33.950153, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8619> 2014-11-08 18:14:34.134169 7fc5a39d2700 5 -- op tracker -- seq: 46794, time: 2014-11-08 18:14:34.134044, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8618> 2014-11-08 18:14:34.134191 7fc5a39d2700 5 -- op tracker -- seq: 46794, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8617> 2014-11-08 18:14:34.134246 7fc5af314700 5 -- op tracker -- seq: 46794, time: 2014-11-08 18:14:34.134246, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8616> 2014-11-08 18:14:34.141754 7fc5a39d2700 5 -- op tracker -- seq: 46794, time: 2014-11-08 18:14:34.141753, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(fababddd/10000029824.0000001c/head//4, version: 3033'8706, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fababddd/10000029824.0000001c/head//4@3033'8706, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c0dabddd/1000001aa14.00000124/head//4, version: 2829'4658, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c0dabddd/1000001aa14.00000124/head//4@2829'4658, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(881bbddd/1000001b509.00000175/head//4, version: 3033'6259, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(881bbddd/1000001b509.00000175/head//4@3033'6259, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b1bbddd/1000002baa8.000000b5/head//4, version: 3473'14366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 8b1bbddd/1000002baa8.000000b5/head//4@3473'14366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8615> 2014-11-08 18:14:34.196940 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8470 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) v2 ==== 110+0+0 (3663019312 0 0) 0x229b9200 con 0x1c5c0580 -8614> 2014-11-08 18:14:34.196970 7fc5a152e700 5 -- op tracker -- seq: 46795, time: 2014-11-08 18:14:34.196895, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) -8613> 2014-11-08 18:14:34.196979 7fc5a152e700 5 -- op tracker -- seq: 46795, time: 2014-11-08 18:14:34.196896, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) -8612> 2014-11-08 18:14:34.196982 7fc5a152e700 5 -- op tracker -- seq: 46795, time: 2014-11-08 18:14:34.196938, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) -8611> 2014-11-08 18:14:34.196984 7fc5a152e700 5 -- op tracker -- seq: 46795, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) -8610> 2014-11-08 18:14:34.197031 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8469 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x24959860 con 0x1c5c0580 -8609> 2014-11-08 18:14:34.197041 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.196808, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8608> 2014-11-08 18:14:34.197045 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.196810, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8607> 2014-11-08 18:14:34.197047 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.196878, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8606> 2014-11-08 18:14:34.197050 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.197038, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8605> 2014-11-08 18:14:34.197053 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.197053, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8604> 2014-11-08 18:14:34.197059 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.197059, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8603> 2014-11-08 18:14:34.197064 7fc5b9b29700 5 -- op tracker -- seq: 46796, time: 2014-11-08 18:14:34.197064, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8602> 2014-11-08 18:14:34.197096 7fc5ad310700 5 -- op tracker -- seq: 46795, time: 2014-11-08 18:14:34.197096, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) -8601> 2014-11-08 18:14:34.197167 7fc5ad310700 5 -- op tracker -- seq: 46795, time: 2014-11-08 18:14:34.197167, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(f3ffe343/1000002b4e5.0000002b/head//4)]) -8600> 2014-11-08 18:14:34.197440 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=5d90f343/1000002b421.0000001e/head//4)) v1 -- ?+0 0x21a54140 con 0x1e50aec0 -8599> 2014-11-08 18:14:34.197458 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x21a55580 con 0x42c86e0 -8598> 2014-11-08 18:14:34.197468 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is b931f343/1000001a121.0000017c/head//4 -8597> 2014-11-08 18:14:34.197488 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb f3ffe343/1000002b4e5.0000002b/head//4) v3 -- ?+0 0x1d339f80 con 0x1c5c0580 -8596> 2014-11-08 18:14:34.197554 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x21a55580 con 0x42c86e0 -8595> 2014-11-08 18:14:34.197564 7fc5b9328700 5 -- op tracker -- seq: 46797, time: 2014-11-08 18:14:34.197464, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) -8594> 2014-11-08 18:14:34.197568 7fc5b9328700 5 -- op tracker -- seq: 46797, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) -8593> 2014-11-08 18:14:34.197571 7fc5b9328700 5 -- op tracker -- seq: 46797, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) -8592> 2014-11-08 18:14:34.197574 7fc5b9328700 5 -- op tracker -- seq: 46797, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) -8591> 2014-11-08 18:14:34.197598 7fc5afb15700 5 -- op tracker -- seq: 46797, time: 2014-11-08 18:14:34.197598, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) -8590> 2014-11-08 18:14:34.260758 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) v1 -- ?+0 0x211d7d00 con 0x42c86e0 -8589> 2014-11-08 18:14:34.260803 7fc5afb15700 5 -- op tracker -- seq: 46797, time: 2014-11-08 18:14:34.260803, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4397, to_read={5d90f343/1000002b421.0000001e/head//4=0,4194304}, attrs_to_read=)) -8588> 2014-11-08 18:14:34.260810 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d7d00 con 0x42c86e0 -8587> 2014-11-08 18:14:34.260825 7fc5b9328700 5 -- op tracker -- seq: 46798, time: 2014-11-08 18:14:34.260793, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) -8586> 2014-11-08 18:14:34.260828 7fc5b9328700 5 -- op tracker -- seq: 46798, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) -8585> 2014-11-08 18:14:34.260830 7fc5b9328700 5 -- op tracker -- seq: 46798, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) -8584> 2014-11-08 18:14:34.260832 7fc5b9328700 5 -- op tracker -- seq: 46798, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) -8583> 2014-11-08 18:14:34.260863 7fc5ad310700 5 -- op tracker -- seq: 46798, time: 2014-11-08 18:14:34.260863, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) -8582> 2014-11-08 18:14:34.260903 7fc5ad310700 5 -- op tracker -- seq: 46798, time: 2014-11-08 18:14:34.260903, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=0)) -8581> 2014-11-08 18:14:34.281053 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8471 ==== MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1883203956 0 0) 0x229bd000 con 0x1c5c0580 -8580> 2014-11-08 18:14:34.281090 7fc5a152e700 5 -- op tracker -- seq: 46799, time: 2014-11-08 18:14:34.256242, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8579> 2014-11-08 18:14:34.281103 7fc5a152e700 5 -- op tracker -- seq: 46799, time: 2014-11-08 18:14:34.256244, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8578> 2014-11-08 18:14:34.281111 7fc5a152e700 5 -- op tracker -- seq: 46799, time: 2014-11-08 18:14:34.281044, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8577> 2014-11-08 18:14:34.281119 7fc5a152e700 5 -- op tracker -- seq: 46799, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8576> 2014-11-08 18:14:34.281158 7fc5b0316700 5 -- op tracker -- seq: 46799, time: 2014-11-08 18:14:34.281158, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8575> 2014-11-08 18:14:34.281234 7fc5b0316700 5 -- op tracker -- seq: 46799, time: 2014-11-08 18:14:34.281234, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(fdf5fcb5/1000002a57f.00000021/head//4, version: 3033'5022, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fdf5fcb5/1000002a57f.00000021/head//4@3033'5022, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8574> 2014-11-08 18:14:34.312871 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3731 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:34.311754) v2 ==== 47+0+0 (21697668 0 0) 0x22868d20 con 0x1cce3700 -8573> 2014-11-08 18:14:34.312898 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.311754) v2 -- ?+0 0x2336f620 con 0x1cce3700 -8572> 2014-11-08 18:14:34.312918 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3731 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:34.311754) v2 ==== 47+0+0 (21697668 0 0) 0x23686720 con 0x1cce3180 -8571> 2014-11-08 18:14:34.312927 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.311754) v2 -- ?+0 0x1d041680 con 0x1cce3180 -8570> 2014-11-08 18:14:34.438370 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1572 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2694488650 0 0) 0x22879680 con 0x21468c60 -8569> 2014-11-08 18:14:34.438394 7fc5a39d2700 5 -- op tracker -- seq: 46800, time: 2014-11-08 18:14:34.438278, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) -8568> 2014-11-08 18:14:34.438404 7fc5a39d2700 5 -- op tracker -- seq: 46800, time: 2014-11-08 18:14:34.438280, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) -8567> 2014-11-08 18:14:34.438411 7fc5a39d2700 5 -- op tracker -- seq: 46800, time: 2014-11-08 18:14:34.438363, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) -8566> 2014-11-08 18:14:34.438418 7fc5a39d2700 5 -- op tracker -- seq: 46800, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) -8565> 2014-11-08 18:14:34.438460 7fc5acb0f700 5 -- op tracker -- seq: 46800, time: 2014-11-08 18:14:34.438460, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) -8564> 2014-11-08 18:14:34.493937 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2177 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) v1 ==== 2097696+0+0 (3885882967 0 0) 0x211d6e00 con 0x1e50aec0 -8563> 2014-11-08 18:14:34.493980 7fc5a56ef700 5 -- op tracker -- seq: 46801, time: 2014-11-08 18:14:34.355898, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) -8562> 2014-11-08 18:14:34.493990 7fc5a56ef700 5 -- op tracker -- seq: 46801, time: 2014-11-08 18:14:34.355900, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) -8561> 2014-11-08 18:14:34.493993 7fc5a56ef700 5 -- op tracker -- seq: 46801, time: 2014-11-08 18:14:34.493911, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) -8560> 2014-11-08 18:14:34.493997 7fc5a56ef700 5 -- op tracker -- seq: 46801, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) -8559> 2014-11-08 18:14:34.494033 7fc5afb15700 5 -- op tracker -- seq: 46801, time: 2014-11-08 18:14:34.494033, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) -8558> 2014-11-08 18:14:34.498005 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(5d90f343/1000002b421.0000001e/head//4, version: 3468'12506, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5d90f343/1000002b421.0000001e/head//4@3468'12506, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3aba00 con 0x1c5c0580 -8557> 2014-11-08 18:14:34.498102 7fc5afb15700 5 -- op tracker -- seq: 46801, time: 2014-11-08 18:14:34.498102, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4397, attrs_read=1)) -8556> 2014-11-08 18:14:34.503879 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x2495ecc0 con 0x21468c60 -8555> 2014-11-08 18:14:34.504088 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x249592c0 con 0x21468c60 -8554> 2014-11-08 18:14:34.504132 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x24958b40 con 0x21468c60 -8553> 2014-11-08 18:14:34.504162 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1ec24ce0 con 0x21468c60 -8552> 2014-11-08 18:14:34.504284 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(fababddd/10000029824.0000001c/head//4),PushReplyOp(c0dabddd/1000001aa14.00000124/head//4),PushReplyOp(881bbddd/1000001b509.00000175/head//4),PushReplyOp(8b1bbddd/1000002baa8.000000b5/head//4)]) v2 -- ?+0 0x21610000 con 0x21468c60 -8551> 2014-11-08 18:14:34.527017 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59230, attrs_read=0)) v1 -- ?+0 0x21838000 con 0x21468c60 -8550> 2014-11-08 18:14:34.527074 7fc5acb0f700 5 -- op tracker -- seq: 46800, time: 2014-11-08 18:14:34.527074, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59230, to_read={aa3e314e/10000019a15.00000084/head//4=0,4194304}, attrs_to_read=)) -8549> 2014-11-08 18:14:34.577654 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2257e900 con 0x1c5c0580 -8548> 2014-11-08 18:14:34.577739 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(fdf5fcb5/1000002a57f.00000021/head//4)]) v2 -- ?+0 0x1d5b4c00 con 0x1c5c0580 -8547> 2014-11-08 18:14:34.616343 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8472 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x2257e900 con 0x1c5c0580 -8546> 2014-11-08 18:14:34.616378 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616256, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8545> 2014-11-08 18:14:34.616376 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8473 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) v2 ==== 110+0+0 (2524776986 0 0) 0x1d5b4c00 con 0x1c5c0580 -8544> 2014-11-08 18:14:34.616387 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616257, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8543> 2014-11-08 18:14:34.616391 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616321, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8542> 2014-11-08 18:14:34.616389 7fc5a152e700 5 -- op tracker -- seq: 46803, time: 2014-11-08 18:14:34.616341, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) -8541> 2014-11-08 18:14:34.616394 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616373, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8540> 2014-11-08 18:14:34.616394 7fc5a152e700 5 -- op tracker -- seq: 46803, time: 2014-11-08 18:14:34.616341, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) -8539> 2014-11-08 18:14:34.616397 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616397, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8538> 2014-11-08 18:14:34.616398 7fc5a152e700 5 -- op tracker -- seq: 46803, time: 2014-11-08 18:14:34.616375, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) -8537> 2014-11-08 18:14:34.616401 7fc5a152e700 5 -- op tracker -- seq: 46803, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) -8536> 2014-11-08 18:14:34.616406 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616406, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8535> 2014-11-08 18:14:34.616413 7fc5b9b29700 5 -- op tracker -- seq: 46802, time: 2014-11-08 18:14:34.616412, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8534> 2014-11-08 18:14:34.616493 7fc5ad310700 5 -- op tracker -- seq: 46803, time: 2014-11-08 18:14:34.616493, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) -8533> 2014-11-08 18:14:34.616582 7fc5ad310700 5 -- op tracker -- seq: 46803, time: 2014-11-08 18:14:34.616582, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(5d90f343/1000002b421.0000001e/head//4)]) -8532> 2014-11-08 18:14:34.616898 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=b931f343/1000001a121.0000017c/head//4)) v1 -- ?+0 0x21a57bc0 con 0x1e50aec0 -8531> 2014-11-08 18:14:34.616921 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x21a557c0 con 0x42c86e0 -8530> 2014-11-08 18:14:34.616937 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x21a557c0 con 0x42c86e0 -8529> 2014-11-08 18:14:34.616934 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=1 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 5981f343/1000002ab09.00000020/head//4 -8528> 2014-11-08 18:14:34.616949 7fc5b9328700 5 -- op tracker -- seq: 46804, time: 2014-11-08 18:14:34.616930, event: header_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) -8527> 2014-11-08 18:14:34.616959 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5773/5773 lb 5d90f343/1000002b421.0000001e/head//4) v3 -- ?+0 0x1d655100 con 0x1c5c0580 -8526> 2014-11-08 18:14:34.616954 7fc5b9328700 5 -- op tracker -- seq: 46804, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) -8525> 2014-11-08 18:14:34.616997 7fc5b9328700 5 -- op tracker -- seq: 46804, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) -8524> 2014-11-08 18:14:34.616999 7fc5b9328700 5 -- op tracker -- seq: 46804, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) -8523> 2014-11-08 18:14:34.617037 7fc5afb15700 5 -- op tracker -- seq: 46804, time: 2014-11-08 18:14:34.617037, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) -8522> 2014-11-08 18:14:34.617144 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2178 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) v2 ==== 110+0+0 (1596892331 0 0) 0x1d5b1000 con 0x1e50aec0 -8521> 2014-11-08 18:14:34.617157 7fc5a56ef700 5 -- op tracker -- seq: 46805, time: 2014-11-08 18:14:34.617100, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) -8520> 2014-11-08 18:14:34.617162 7fc5a56ef700 5 -- op tracker -- seq: 46805, time: 2014-11-08 18:14:34.617102, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) -8519> 2014-11-08 18:14:34.617165 7fc5a56ef700 5 -- op tracker -- seq: 46805, time: 2014-11-08 18:14:34.617140, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) -8518> 2014-11-08 18:14:34.617169 7fc5a56ef700 5 -- op tracker -- seq: 46805, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) -8517> 2014-11-08 18:14:34.617193 7fc5adb11700 5 -- op tracker -- seq: 46805, time: 2014-11-08 18:14:34.617193, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) -8516> 2014-11-08 18:14:34.617233 7fc5adb11700 5 -- op tracker -- seq: 46805, time: 2014-11-08 18:14:34.617233, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(885c1d83/1000002b519.00000021/head//4)]) -8515> 2014-11-08 18:14:34.622002 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=adfc1d83/1000002a6c6.00000008/head//4)) v1 -- ?+0 0x22497bc0 con 0x1c5c0580 -8514> 2014-11-08 18:14:34.622021 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x22490900 con 0x42c86e0 -8513> 2014-11-08 18:14:34.622031 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 8c4e1d83/1000001f006.00000000/head//4 -8512> 2014-11-08 18:14:34.622062 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb 885c1d83/1000002b519.00000021/head//4) v3 -- ?+0 0x1d650900 con 0x1e50aec0 -8511> 2014-11-08 18:14:34.622086 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x22490900 con 0x42c86e0 -8510> 2014-11-08 18:14:34.622096 7fc5b9328700 5 -- op tracker -- seq: 46806, time: 2014-11-08 18:14:34.622027, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) -8509> 2014-11-08 18:14:34.622101 7fc5b9328700 5 -- op tracker -- seq: 46806, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) -8508> 2014-11-08 18:14:34.622104 7fc5b9328700 5 -- op tracker -- seq: 46806, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) -8507> 2014-11-08 18:14:34.622106 7fc5b9328700 5 -- op tracker -- seq: 46806, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) -8506> 2014-11-08 18:14:34.622132 7fc5b0316700 5 -- op tracker -- seq: 46806, time: 2014-11-08 18:14:34.622131, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) -8505> 2014-11-08 18:14:34.662053 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) v1 -- ?+0 0x211d6e00 con 0x42c86e0 -8504> 2014-11-08 18:14:34.662112 7fc5afb15700 5 -- op tracker -- seq: 46804, time: 2014-11-08 18:14:34.662111, event: done, op: MOSDECSubOpRead(4.43s0 5773 ECSubRead(tid=4398, to_read={b931f343/1000001a121.0000017c/head//4=0,4194304}, attrs_to_read=)) -8503> 2014-11-08 18:14:34.662115 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d6e00 con 0x42c86e0 -8502> 2014-11-08 18:14:34.662138 7fc5b9328700 5 -- op tracker -- seq: 46807, time: 2014-11-08 18:14:34.662100, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) -8501> 2014-11-08 18:14:34.662144 7fc5b9328700 5 -- op tracker -- seq: 46807, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) -8500> 2014-11-08 18:14:34.662148 7fc5b9328700 5 -- op tracker -- seq: 46807, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) -8499> 2014-11-08 18:14:34.662151 7fc5b9328700 5 -- op tracker -- seq: 46807, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) -8498> 2014-11-08 18:14:34.662206 7fc5ad310700 5 -- op tracker -- seq: 46807, time: 2014-11-08 18:14:34.662206, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) -8497> 2014-11-08 18:14:34.662247 7fc5ad310700 5 -- op tracker -- seq: 46807, time: 2014-11-08 18:14:34.662246, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=0)) -8496> 2014-11-08 18:14:34.670886 7fc5c362f700 5 osd.9 5773 tick -8495> 2014-11-08 18:14:34.684637 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) v1 -- ?+0 0x224bdf00 con 0x42c86e0 -8494> 2014-11-08 18:14:34.684663 7fc5b0316700 5 -- op tracker -- seq: 46806, time: 2014-11-08 18:14:34.684663, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4399, to_read={adfc1d83/1000002a6c6.00000008/head//4=0,4194304}, attrs_to_read=)) -8493> 2014-11-08 18:14:34.684668 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224bdf00 con 0x42c86e0 -8492> 2014-11-08 18:14:34.684679 7fc5b9328700 5 -- op tracker -- seq: 46808, time: 2014-11-08 18:14:34.684656, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) -8491> 2014-11-08 18:14:34.684683 7fc5b9328700 5 -- op tracker -- seq: 46808, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) -8490> 2014-11-08 18:14:34.684693 7fc5b9328700 5 -- op tracker -- seq: 46808, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) -8489> 2014-11-08 18:14:34.684695 7fc5b9328700 5 -- op tracker -- seq: 46808, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) -8488> 2014-11-08 18:14:34.684718 7fc5adb11700 5 -- op tracker -- seq: 46808, time: 2014-11-08 18:14:34.684718, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) -8487> 2014-11-08 18:14:34.684747 7fc5adb11700 5 -- op tracker -- seq: 46808, time: 2014-11-08 18:14:34.684747, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=0)) -8486> 2014-11-08 18:14:34.797062 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2179 ==== MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) v1 ==== 2097696+0+0 (833719946 0 0) 0x211d1180 con 0x1e50aec0 -8485> 2014-11-08 18:14:34.797095 7fc5a56ef700 5 -- op tracker -- seq: 46809, time: 2014-11-08 18:14:34.758888, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) -8484> 2014-11-08 18:14:34.797104 7fc5a56ef700 5 -- op tracker -- seq: 46809, time: 2014-11-08 18:14:34.758891, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) -8483> 2014-11-08 18:14:34.797107 7fc5a56ef700 5 -- op tracker -- seq: 46809, time: 2014-11-08 18:14:34.797054, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) -8482> 2014-11-08 18:14:34.797110 7fc5a56ef700 5 -- op tracker -- seq: 46809, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) -8481> 2014-11-08 18:14:34.797148 7fc5afb15700 5 -- op tracker -- seq: 46809, time: 2014-11-08 18:14:34.797148, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) -8480> 2014-11-08 18:14:34.800901 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5773 [PushOp(b931f343/1000001a121.0000017c/head//4, version: 397'3398, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b931f343/1000001a121.0000017c/head//4@397'3398, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d5b1800 con 0x1c5c0580 -8479> 2014-11-08 18:14:34.800965 7fc5afb15700 5 -- op tracker -- seq: 46809, time: 2014-11-08 18:14:34.800965, event: done, op: MOSDECSubOpReadReply(4.43s0 5773 ECSubReadReply(tid=4398, attrs_read=1)) -8478> 2014-11-08 18:14:34.813056 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3732 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:34.812412) v2 ==== 47+0+0 (3529532813 0 0) 0x2286f440 con 0x1cce3700 -8477> 2014-11-08 18:14:34.813084 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.812412) v2 -- ?+0 0x22868d20 con 0x1cce3700 -8476> 2014-11-08 18:14:34.813103 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3732 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:34.812412) v2 ==== 47+0+0 (3529532813 0 0) 0x23685a00 con 0x1cce3180 -8475> 2014-11-08 18:14:34.813113 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.812412) v2 -- ?+0 0x23686720 con 0x1cce3180 -8474> 2014-11-08 18:14:34.840824 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -8473> 2014-11-08 18:14:34.840849 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x22454920 con 0x1cae2940 -8472> 2014-11-08 18:14:34.840870 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x22e992c0 con 0x1cae2680 -8471> 2014-11-08 18:14:34.840881 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x22e9f620 con 0x1c5c02c0 -8470> 2014-11-08 18:14:34.840891 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x22e98780 con 0x1cae0160 -8469> 2014-11-08 18:14:34.840901 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x22e9cce0 con 0x1cae0f20 -8468> 2014-11-08 18:14:34.840911 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x2026f800 con 0x1cae0c60 -8467> 2014-11-08 18:14:34.840921 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x1ec27440 con 0x1cae23c0 -8466> 2014-11-08 18:14:34.840931 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x1caae540 con 0x1cae56a0 -8465> 2014-11-08 18:14:34.840941 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x211e10e0 con 0x1cbf9ce0 -8464> 2014-11-08 18:14:34.840951 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x2026e540 con 0x1d603020 -8463> 2014-11-08 18:14:34.840961 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x21221e00 con 0x1dbd9b80 -8462> 2014-11-08 18:14:34.840971 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x21223de0 con 0x1d235d80 -8461> 2014-11-08 18:14:34.841048 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x22a232a0 con 0x1cae1fa0 -8460> 2014-11-08 18:14:34.841059 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x1ddccce0 con 0x1cae1ce0 -8459> 2014-11-08 18:14:34.841069 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x1ddc92c0 con 0x1cae1340 -8458> 2014-11-08 18:14:34.841078 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x1ddca760 con 0x1cae1080 -8457> 2014-11-08 18:14:34.841087 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x224512c0 con 0x1cae1a20 -8456> 2014-11-08 18:14:34.841096 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:34.840846) v2 -- ?+0 0x233fb2a0 con 0x1cae2c00 -8455> 2014-11-08 18:14:34.842112 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1d4f01e0 con 0x1cae2940 -8454> 2014-11-08 18:14:34.842133 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1f4c2ee0 con 0x1cae2680 -8453> 2014-11-08 18:14:34.842145 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 303 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1d326720 con 0x1d235d80 -8452> 2014-11-08 18:14:34.842157 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x22456900 con 0x1cae1340 -8451> 2014-11-08 18:14:34.842169 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1decc920 con 0x1cae1080 -8450> 2014-11-08 18:14:34.842180 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x25fd2b20 con 0x1cae1fa0 -8449> 2014-11-08 18:14:34.842192 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x4537260 con 0x1cae1ce0 -8448> 2014-11-08 18:14:34.842328 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1d322760 con 0x1cae1a20 -8447> 2014-11-08 18:14:34.842343 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1dbf8780 con 0x1cae2c00 -8446> 2014-11-08 18:14:34.842354 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 303 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1db1bde0 con 0x1dbd9b80 -8445> 2014-11-08 18:14:34.842364 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x257c4b00 con 0x1c5c02c0 -8444> 2014-11-08 18:14:34.842376 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x2290fbc0 con 0x1cae0160 -8443> 2014-11-08 18:14:34.842632 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x2515c380 con 0x1cae23c0 -8442> 2014-11-08 18:14:34.842646 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x2583c920 con 0x1cae56a0 -8441> 2014-11-08 18:14:34.842878 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8474 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) v1 ==== 2097696+0+0 (999013069 0 0) 0x224beb80 con 0x1c5c0580 -8440> 2014-11-08 18:14:34.842892 7fc5a152e700 5 -- op tracker -- seq: 46810, time: 2014-11-08 18:14:34.716250, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) -8439> 2014-11-08 18:14:34.842898 7fc5a152e700 5 -- op tracker -- seq: 46810, time: 2014-11-08 18:14:34.716252, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) -8438> 2014-11-08 18:14:34.842901 7fc5a152e700 5 -- op tracker -- seq: 46810, time: 2014-11-08 18:14:34.842873, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) -8437> 2014-11-08 18:14:34.842904 7fc5a152e700 5 -- op tracker -- seq: 46810, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) -8436> 2014-11-08 18:14:34.842937 7fc5b0316700 5 -- op tracker -- seq: 46810, time: 2014-11-08 18:14:34.842937, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) -8435> 2014-11-08 18:14:34.842960 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8475 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) v3 ==== 757+0+0 (298500370 0 0) 0x1d655100 con 0x1c5c0580 -8434> 2014-11-08 18:14:34.842970 7fc5a152e700 5 -- op tracker -- seq: 46811, time: 2014-11-08 18:14:34.842928, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8433> 2014-11-08 18:14:34.842973 7fc5a152e700 5 -- op tracker -- seq: 46811, time: 2014-11-08 18:14:34.842929, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8432> 2014-11-08 18:14:34.842977 7fc5a152e700 5 -- op tracker -- seq: 46811, time: 2014-11-08 18:14:34.842957, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8431> 2014-11-08 18:14:34.842980 7fc5a152e700 5 -- op tracker -- seq: 46811, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8430> 2014-11-08 18:14:34.843877 7fc5adb11700 5 -- op tracker -- seq: 46811, time: 2014-11-08 18:14:34.843877, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8429> 2014-11-08 18:14:34.843888 7fc5adb11700 5 -- op tracker -- seq: 46811, time: 2014-11-08 18:14:34.843887, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8428> 2014-11-08 18:14:34.843933 7fc5adb11700 5 -- op tracker -- seq: 46811, time: 2014-11-08 18:14:34.843933, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fdf5fcb5/1000002a57f.00000021/head//4) -8427> 2014-11-08 18:14:34.846919 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(adfc1d83/1000002a6c6.00000008/head//4, version: 3458'5039, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(adfc1d83/1000002a6c6.00000008/head//4@3458'5039, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229bd000 con 0x1e50aec0 -8426> 2014-11-08 18:14:34.847247 7fc5b0316700 5 -- op tracker -- seq: 46810, time: 2014-11-08 18:14:34.847247, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4399, attrs_read=1)) -8425> 2014-11-08 18:14:34.850466 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 496 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1eae3fc0 con 0x1cbf9ce0 -8424> 2014-11-08 18:14:34.850737 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 496 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1dbfeea0 con 0x1d603020 -8423> 2014-11-08 18:14:34.876557 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1573 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) v3 ==== 757+0+0 (2039667285 0 0) 0x1e58c800 con 0x21468c60 -8422> 2014-11-08 18:14:34.876581 7fc5a39d2700 5 -- op tracker -- seq: 46812, time: 2014-11-08 18:14:34.876485, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8421> 2014-11-08 18:14:34.876588 7fc5a39d2700 5 -- op tracker -- seq: 46812, time: 2014-11-08 18:14:34.876488, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8420> 2014-11-08 18:14:34.876592 7fc5a39d2700 5 -- op tracker -- seq: 46812, time: 2014-11-08 18:14:34.876551, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8419> 2014-11-08 18:14:34.876596 7fc5a39d2700 5 -- op tracker -- seq: 46812, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8418> 2014-11-08 18:14:34.876625 7fc5af314700 5 -- op tracker -- seq: 46812, time: 2014-11-08 18:14:34.876625, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8417> 2014-11-08 18:14:34.876636 7fc5af314700 5 -- op tracker -- seq: 46812, time: 2014-11-08 18:14:34.876636, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8416> 2014-11-08 18:14:34.876703 7fc5af314700 5 -- op tracker -- seq: 46812, time: 2014-11-08 18:14:34.876703, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 8b1bbddd/1000002baa8.000000b5/head//4) -8415> 2014-11-08 18:14:34.976862 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2180 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) v2 ==== 370+0+0 (3308357366 0 0) 0x229bd000 con 0x1e50aec0 -8414> 2014-11-08 18:14:34.976899 7fc5a56ef700 5 -- op tracker -- seq: 46813, time: 2014-11-08 18:14:34.976682, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) -8413> 2014-11-08 18:14:34.976910 7fc5a56ef700 5 -- op tracker -- seq: 46813, time: 2014-11-08 18:14:34.976733, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) -8412> 2014-11-08 18:14:34.976916 7fc5a56ef700 5 -- op tracker -- seq: 46813, time: 2014-11-08 18:14:34.976855, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) -8411> 2014-11-08 18:14:34.976923 7fc5a56ef700 5 -- op tracker -- seq: 46813, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) -8410> 2014-11-08 18:14:34.976962 7fc5ac30e700 5 -- op tracker -- seq: 46813, time: 2014-11-08 18:14:34.976961, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) -8409> 2014-11-08 18:14:34.977099 7fc5ac30e700 5 -- op tracker -- seq: 46813, time: 2014-11-08 18:14:34.977099, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(472bf418/10000029051.0000000f/head//4),PushReplyOp(fe9bf418/1000001a0b9.0000068e/head//4),PushReplyOp(af1cf418/1000002b54f.0000016e/head//4),PushReplyOp(b1bcf418/1000001ac2f.0000004f/head//4),PushReplyOp(700df418/10000025f2d.00000000/head//4)]) -8408> 2014-11-08 18:14:34.977864 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2181 ==== MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) v2 ==== 175+0+0 (2372446159 0 0) 0x1c6a7c00 con 0x1e50aec0 -8407> 2014-11-08 18:14:34.977876 7fc5a56ef700 5 -- op tracker -- seq: 46814, time: 2014-11-08 18:14:34.977822, event: header_read, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) -8406> 2014-11-08 18:14:34.977882 7fc5a56ef700 5 -- op tracker -- seq: 46814, time: 2014-11-08 18:14:34.977823, event: throttled, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) -8405> 2014-11-08 18:14:34.977887 7fc5a56ef700 5 -- op tracker -- seq: 46814, time: 2014-11-08 18:14:34.977861, event: all_read, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) -8404> 2014-11-08 18:14:34.977892 7fc5a56ef700 5 -- op tracker -- seq: 46814, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) -8403> 2014-11-08 18:14:34.977915 7fc5adb11700 5 -- op tracker -- seq: 46814, time: 2014-11-08 18:14:34.977915, event: reached_pg, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) -8402> 2014-11-08 18:14:34.977963 7fc5adb11700 5 -- op tracker -- seq: 46814, time: 2014-11-08 18:14:34.977963, event: done, op: MOSDPGPushReply(4.1as0 5773 [PushReplyOp(891b231a/100000199b7.00000367/head//4),PushReplyOp(e9ab231a/1000002a767.00000093/head//4)]) -8401> 2014-11-08 18:14:35.007537 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8476 ==== pg_trim(4.43 to 5734'16291 e5773) v1 ==== 34+0+0 (3914395777 0 0) 0x22d505a0 con 0x1c5c0580 -8400> 2014-11-08 18:14:35.007566 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8477 ==== MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) v2 ==== 110+0+0 (1767693492 0 0) 0x1d5b1800 con 0x1c5c0580 -8399> 2014-11-08 18:14:35.007570 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007456, event: header_read, op: pg_trim(4.43 to 5734'16291 e5773) -8398> 2014-11-08 18:14:35.007577 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007459, event: throttled, op: pg_trim(4.43 to 5734'16291 e5773) -8397> 2014-11-08 18:14:35.007577 7fc5a152e700 5 -- op tracker -- seq: 46816, time: 2014-11-08 18:14:35.007535, event: header_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) -8396> 2014-11-08 18:14:35.007580 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007518, event: all_read, op: pg_trim(4.43 to 5734'16291 e5773) -8395> 2014-11-08 18:14:35.007582 7fc5a152e700 5 -- op tracker -- seq: 46816, time: 2014-11-08 18:14:35.007535, event: throttled, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) -8394> 2014-11-08 18:14:35.007583 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007565, event: dispatched, op: pg_trim(4.43 to 5734'16291 e5773) -8393> 2014-11-08 18:14:35.007586 7fc5a152e700 5 -- op tracker -- seq: 46816, time: 2014-11-08 18:14:35.007565, event: all_read, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) -8392> 2014-11-08 18:14:35.007587 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007587, event: waiting_for_osdmap, op: pg_trim(4.43 to 5734'16291 e5773) -8391> 2014-11-08 18:14:35.007589 7fc5a152e700 5 -- op tracker -- seq: 46816, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) -8390> 2014-11-08 18:14:35.007596 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007596, event: started, op: pg_trim(4.43 to 5734'16291 e5773) -8389> 2014-11-08 18:14:35.007601 7fc5b9b29700 5 -- op tracker -- seq: 46815, time: 2014-11-08 18:14:35.007601, event: done, op: pg_trim(4.43 to 5734'16291 e5773) -8388> 2014-11-08 18:14:35.007658 7fc5ad310700 5 -- op tracker -- seq: 46816, time: 2014-11-08 18:14:35.007658, event: reached_pg, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) -8387> 2014-11-08 18:14:35.007730 7fc5a152e700 5 -- op tracker -- seq: 46816, time: 2014-11-08 18:14:35.007730, event: done, op: MOSDPGPushReply(4.43s0 5773 [PushReplyOp(b931f343/1000001a121.0000017c/head//4)]) -8386> 2014-11-08 18:14:35.025338 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1574 ==== MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (422457729 0 0) 0x21610000 con 0x21468c60 -8385> 2014-11-08 18:14:35.025385 7fc5a39d2700 5 -- op tracker -- seq: 46817, time: 2014-11-08 18:14:34.982665, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8384> 2014-11-08 18:14:35.025403 7fc5a39d2700 5 -- op tracker -- seq: 46817, time: 2014-11-08 18:14:34.982667, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8383> 2014-11-08 18:14:35.025415 7fc5a39d2700 5 -- op tracker -- seq: 46817, time: 2014-11-08 18:14:35.025331, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8382> 2014-11-08 18:14:35.025428 7fc5a39d2700 5 -- op tracker -- seq: 46817, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8381> 2014-11-08 18:14:35.025489 7fc5acb0f700 5 -- op tracker -- seq: 46817, time: 2014-11-08 18:14:35.025489, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8380> 2014-11-08 18:14:35.025574 7fc5a39d2700 5 -- op tracker -- seq: 46817, time: 2014-11-08 18:14:35.025574, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(629bbddd/1000001a794.000001d6/head//4, version: 619'4217, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(629bbddd/1000001a794.000001d6/head//4@619'4217, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8379> 2014-11-08 18:14:35.027165 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2182 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) v2 ==== 110+0+0 (2058755280 0 0) 0x2393b400 con 0x1e50aec0 -8378> 2014-11-08 18:14:35.027185 7fc5a56ef700 5 -- op tracker -- seq: 46818, time: 2014-11-08 18:14:35.027108, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8377> 2014-11-08 18:14:35.027196 7fc5a56ef700 5 -- op tracker -- seq: 46818, time: 2014-11-08 18:14:35.027109, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8376> 2014-11-08 18:14:35.027206 7fc5a56ef700 5 -- op tracker -- seq: 46818, time: 2014-11-08 18:14:35.027160, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8375> 2014-11-08 18:14:35.027215 7fc5a56ef700 5 -- op tracker -- seq: 46818, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8374> 2014-11-08 18:14:35.027246 7fc5aeb13700 5 -- op tracker -- seq: 46818, time: 2014-11-08 18:14:35.027246, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8373> 2014-11-08 18:14:35.027307 7fc5a56ef700 5 -- op tracker -- seq: 46818, time: 2014-11-08 18:14:35.027307, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(b1b7dd13/10000019aef.00000427/head//4)]) -8372> 2014-11-08 18:14:35.047570 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=ca5df418/1000001b502.00000378/head//4,246df418/1000002bdbc.000000fe/head//4,611ef418/1000002a661.00000011/head//4,8a2ef418/10000019d91.00000026/head//4,318ef418/1000002a454.00000037/head//4)) v1 -- ?+0 0x22491b00 con 0x1c5c0580 -8371> 2014-11-08 18:14:35.047614 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x22491440 con 0x42c86e0 -8370> 2014-11-08 18:14:35.047632 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 2ddff418/1000002845d.00000061/head//4 -8369> 2014-11-08 18:14:35.047663 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb 700df418/10000025f2d.00000000/head//4) v3 -- ?+0 0x248b7980 con 0x1e50aec0 -8368> 2014-11-08 18:14:35.047791 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x22491440 con 0x42c86e0 -8367> 2014-11-08 18:14:35.047811 7fc5b9328700 5 -- op tracker -- seq: 46819, time: 2014-11-08 18:14:35.047625, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) -8366> 2014-11-08 18:14:35.047822 7fc5b9328700 5 -- op tracker -- seq: 46819, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) -8365> 2014-11-08 18:14:35.047827 7fc5b9328700 5 -- op tracker -- seq: 46819, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) -8364> 2014-11-08 18:14:35.047836 7fc5b9328700 5 -- op tracker -- seq: 46819, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) -8363> 2014-11-08 18:14:35.047884 7fc5ac30e700 5 -- op tracker -- seq: 46819, time: 2014-11-08 18:14:35.047884, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) -8362> 2014-11-08 18:14:35.063389 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1e893840 con 0x1cae0c60 -8361> 2014-11-08 18:14:35.063448 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3716 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:34.840846) v2 ==== 47+0+0 (4261620685 0 0) 0x1d8b9680 con 0x1cae0f20 -8360> 2014-11-08 18:14:35.118249 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1575 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2541852177 0 0) 0x22879440 con 0x21468c60 -8359> 2014-11-08 18:14:35.118307 7fc5a39d2700 5 -- op tracker -- seq: 46820, time: 2014-11-08 18:14:35.118149, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) -8358> 2014-11-08 18:14:35.118322 7fc5a39d2700 5 -- op tracker -- seq: 46820, time: 2014-11-08 18:14:35.118154, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) -8357> 2014-11-08 18:14:35.118332 7fc5a39d2700 5 -- op tracker -- seq: 46820, time: 2014-11-08 18:14:35.118240, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) -8356> 2014-11-08 18:14:35.118342 7fc5a39d2700 5 -- op tracker -- seq: 46820, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) -8355> 2014-11-08 18:14:35.118416 7fc5af314700 5 -- op tracker -- seq: 46820, time: 2014-11-08 18:14:35.118416, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) -8354> 2014-11-08 18:14:35.120272 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2183 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x1d1b1c20 con 0x1e50aec0 -8353> 2014-11-08 18:14:35.120289 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120214, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -8352> 2014-11-08 18:14:35.120293 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120215, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -8351> 2014-11-08 18:14:35.120296 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120252, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -8350> 2014-11-08 18:14:35.120299 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120285, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -8349> 2014-11-08 18:14:35.120302 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120302, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -8348> 2014-11-08 18:14:35.120308 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120308, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -8347> 2014-11-08 18:14:35.120312 7fc5b9b29700 5 -- op tracker -- seq: 46821, time: 2014-11-08 18:14:35.120312, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -8346> 2014-11-08 18:14:35.135749 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8478 ==== MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1916785160 0 0) 0x1d5b1800 con 0x1c5c0580 -8345> 2014-11-08 18:14:35.135791 7fc5a152e700 5 -- op tracker -- seq: 46822, time: 2014-11-08 18:14:35.049377, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8344> 2014-11-08 18:14:35.135805 7fc5a152e700 5 -- op tracker -- seq: 46822, time: 2014-11-08 18:14:35.049380, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8343> 2014-11-08 18:14:35.135813 7fc5a152e700 5 -- op tracker -- seq: 46822, time: 2014-11-08 18:14:35.135736, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8342> 2014-11-08 18:14:35.135822 7fc5a152e700 5 -- op tracker -- seq: 46822, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8341> 2014-11-08 18:14:35.135866 7fc5b0316700 5 -- op tracker -- seq: 46822, time: 2014-11-08 18:14:35.135865, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8340> 2014-11-08 18:14:35.135964 7fc5b0316700 5 -- op tracker -- seq: 46822, time: 2014-11-08 18:14:35.135964, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(1037fcb5/1000002965c.0000000e/head//4, version: 3033'4216, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1037fcb5/1000002965c.0000000e/head//4@3033'4216, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8339> 2014-11-08 18:14:35.224507 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12044 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2193273751 0 0) 0x1ce61680 con 0x1c5c2680 -8338> 2014-11-08 18:14:35.224542 7fc5a6cfe700 5 -- op tracker -- seq: 46823, time: 2014-11-08 18:14:35.224417, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) -8337> 2014-11-08 18:14:35.224551 7fc5a6cfe700 5 -- op tracker -- seq: 46823, time: 2014-11-08 18:14:35.224420, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) -8336> 2014-11-08 18:14:35.224554 7fc5a6cfe700 5 -- op tracker -- seq: 46823, time: 2014-11-08 18:14:35.224499, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) -8335> 2014-11-08 18:14:35.224557 7fc5a6cfe700 5 -- op tracker -- seq: 46823, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) -8334> 2014-11-08 18:14:35.224610 7fc5aeb13700 5 -- op tracker -- seq: 46823, time: 2014-11-08 18:14:35.224610, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) -8333> 2014-11-08 18:14:35.229093 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=a658dd13/1000001a207.000005b8/head//4,fd19dd13/1000002a5c7.00000003/head//4,da99dd13/1000000e4b7.000003ed/head//4,88aadd13/10000029db8.0000000b/head//4)) v1 -- ?+0 0x224918c0 con 0x1e50aec0 -8332> 2014-11-08 18:14:35.229136 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x224aee40 con 0x42c86e0 -8331> 2014-11-08 18:14:35.229155 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x224aee40 con 0x42c86e0 -8330> 2014-11-08 18:14:35.229154 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=4 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is e71bdd13/1000001a918.00000034/head//4 -8329> 2014-11-08 18:14:35.229174 7fc5b9328700 5 -- op tracker -- seq: 46824, time: 2014-11-08 18:14:35.229146, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) -8328> 2014-11-08 18:14:35.229182 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb b1b7dd13/10000019aef.00000427/head//4) v3 -- ?+0 0x1d652d00 con 0x1c5c0580 -8327> 2014-11-08 18:14:35.229184 7fc5b9328700 5 -- op tracker -- seq: 46824, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) -8326> 2014-11-08 18:14:35.229188 7fc5b9328700 5 -- op tracker -- seq: 46824, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) -8325> 2014-11-08 18:14:35.229193 7fc5b9328700 5 -- op tracker -- seq: 46824, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) -8324> 2014-11-08 18:14:35.229366 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb b1b7dd13/10000019aef.00000427/head//4) v3 -- ?+0 0x1d655e80 con 0x1e50aec0 -8323> 2014-11-08 18:14:35.277982 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59233, attrs_read=0)) v1 -- ?+0 0x1d5fa300 con 0x21468c60 -8322> 2014-11-08 18:14:35.278038 7fc5af314700 5 -- op tracker -- seq: 46820, time: 2014-11-08 18:14:35.278038, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59233, to_read={865e314e/1000001b4d9.0000008f/head//4=0,4194304}, attrs_to_read=)) -8321> 2014-11-08 18:14:35.374022 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2184 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) v2 ==== 240+0+0 (2881868560 0 0) 0x2393b400 con 0x1e50aec0 -8320> 2014-11-08 18:14:35.374060 7fc5a56ef700 5 -- op tracker -- seq: 46825, time: 2014-11-08 18:14:35.373930, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) -8319> 2014-11-08 18:14:35.374070 7fc5a56ef700 5 -- op tracker -- seq: 46825, time: 2014-11-08 18:14:35.373933, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) -8318> 2014-11-08 18:14:35.374074 7fc5a56ef700 5 -- op tracker -- seq: 46825, time: 2014-11-08 18:14:35.374012, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) -8317> 2014-11-08 18:14:35.374078 7fc5a56ef700 5 -- op tracker -- seq: 46825, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) -8316> 2014-11-08 18:14:35.374134 7fc5afb15700 5 -- op tracker -- seq: 46825, time: 2014-11-08 18:14:35.374134, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) -8315> 2014-11-08 18:14:35.374238 7fc5afb15700 5 -- op tracker -- seq: 46825, time: 2014-11-08 18:14:35.374238, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fda9d484/1000001a8ae.000000c9/head//4),PushReplyOp(889bd484/1000002b2cc.00000012/head//4),PushReplyOp(2dabd484/1000002bb6e.00000006/head//4)]) -8314> 2014-11-08 18:14:35.413410 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25182, attrs_read=0)) v1 -- ?+0 0x1d5fd000 con 0x1c5c2680 -8313> 2014-11-08 18:14:35.413531 7fc5aeb13700 5 -- op tracker -- seq: 46823, time: 2014-11-08 18:14:35.413531, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25182, to_read={5ceb3227/1000002b91a.0000003c/head//4=0,4194304}, attrs_to_read=)) -8312> 2014-11-08 18:14:35.413568 7fc5aeb13700 5 -- op tracker -- seq: 46824, time: 2014-11-08 18:14:35.413568, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) -8311> 2014-11-08 18:14:35.451427 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2185 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) v3 ==== 757+0+0 (3739017656 0 0) 0x1d655e80 con 0x1e50aec0 -8310> 2014-11-08 18:14:35.451463 7fc5a56ef700 5 -- op tracker -- seq: 46826, time: 2014-11-08 18:14:35.451326, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8309> 2014-11-08 18:14:35.451474 7fc5a56ef700 5 -- op tracker -- seq: 46826, time: 2014-11-08 18:14:35.451330, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8308> 2014-11-08 18:14:35.451478 7fc5a56ef700 5 -- op tracker -- seq: 46826, time: 2014-11-08 18:14:35.451418, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8307> 2014-11-08 18:14:35.451482 7fc5a56ef700 5 -- op tracker -- seq: 46826, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8306> 2014-11-08 18:14:35.451522 7fc5acb0f700 5 -- op tracker -- seq: 46826, time: 2014-11-08 18:14:35.451522, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8305> 2014-11-08 18:14:35.451537 7fc5acb0f700 5 -- op tracker -- seq: 46826, time: 2014-11-08 18:14:35.451536, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8304> 2014-11-08 18:14:35.451603 7fc5acb0f700 5 -- op tracker -- seq: 46826, time: 2014-11-08 18:14:35.451603, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 435a90b7/1000002b372.0000003a/head//4) -8303> 2014-11-08 18:14:35.478802 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1576 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2320120204 0 0) 0x22879200 con 0x21468c60 -8302> 2014-11-08 18:14:35.478836 7fc5a39d2700 5 -- op tracker -- seq: 46827, time: 2014-11-08 18:14:35.478715, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) -8301> 2014-11-08 18:14:35.478843 7fc5a39d2700 5 -- op tracker -- seq: 46827, time: 2014-11-08 18:14:35.478719, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) -8300> 2014-11-08 18:14:35.478847 7fc5a39d2700 5 -- op tracker -- seq: 46827, time: 2014-11-08 18:14:35.478793, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) -8299> 2014-11-08 18:14:35.478850 7fc5a39d2700 5 -- op tracker -- seq: 46827, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) -8298> 2014-11-08 18:14:35.478898 7fc5af314700 5 -- op tracker -- seq: 46827, time: 2014-11-08 18:14:35.478898, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) -8297> 2014-11-08 18:14:35.499052 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3669 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:35.496285) v2 ==== 47+0+0 (1574332071 0 0) 0x224a50a0 con 0x1cce2ec0 -8296> 2014-11-08 18:14:35.499081 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:35.496285) v2 -- ?+0 0x23685a00 con 0x1cce2ec0 -8295> 2014-11-08 18:14:35.499111 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3669 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:35.496285) v2 ==== 47+0+0 (1574332071 0 0) 0x23141fe0 con 0x1cce3440 -8294> 2014-11-08 18:14:35.499123 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:35.496285) v2 -- ?+0 0x2286f440 con 0x1cce3440 -8293> 2014-11-08 18:14:35.542908 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3703 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:35.540543) v2 ==== 47+0+0 (3724988412 0 0) 0x2336d0a0 con 0x1c8b7d20 -8292> 2014-11-08 18:14:35.542964 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:35.540543) v2 -- ?+0 0x23141fe0 con 0x1c8b7d20 -8291> 2014-11-08 18:14:35.542997 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3703 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:35.540543) v2 ==== 47+0+0 (3724988412 0 0) 0x1d043de0 con 0x1c9f8580 -8290> 2014-11-08 18:14:35.543008 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:35.540543) v2 -- ?+0 0x224a50a0 con 0x1c9f8580 -8289> 2014-11-08 18:14:35.567357 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=78ebd484/1000001a34e.00000036/head//4,586cd484/1000002bca6.0000008e/head//4,516dd484/10000029456.00000006/head//4)) v1 -- ?+0 0x224ad100 con 0x1c5c0580 -8288> 2014-11-08 18:14:35.567395 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x224ad340 con 0x42c86e0 -8287> 2014-11-08 18:14:35.567410 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is fa3ed484/1000002a7de.00000070/head//4 -8286> 2014-11-08 18:14:35.567437 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb 2dabd484/1000002bb6e.00000006/head//4) v3 -- ?+0 0x1d3c8d80 con 0x1e50aec0 -8285> 2014-11-08 18:14:35.567547 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x224ad340 con 0x42c86e0 -8284> 2014-11-08 18:14:35.567563 7fc5b9328700 5 -- op tracker -- seq: 46828, time: 2014-11-08 18:14:35.567405, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) -8283> 2014-11-08 18:14:35.567574 7fc5b9328700 5 -- op tracker -- seq: 46828, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) -8282> 2014-11-08 18:14:35.567579 7fc5b9328700 5 -- op tracker -- seq: 46828, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) -8281> 2014-11-08 18:14:35.567584 7fc5b9328700 5 -- op tracker -- seq: 46828, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) -8280> 2014-11-08 18:14:35.567616 7fc5ad310700 5 -- op tracker -- seq: 46828, time: 2014-11-08 18:14:35.567616, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) -8279> 2014-11-08 18:14:35.591418 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8479 ==== MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9dd5b16/ 1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (2017720941 0 0) 0x1d3aba00 con 0x1c5c0580 -8278> 2014-11-08 18:14:35.591893 7fc5a152e700 5 -- op tracker -- seq: 46829, time: 2014-11-08 18:14:35.297023, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9dd5b16/1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8277> 2014-11-08 18:14:35.591925 7fc5a152e700 5 -- op tracker -- seq: 46829, time: 2014-11-08 18:14:35.297029, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( f9dd5b16/1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8276> 2014-11-08 18:14:35.591951 7fc5a152e700 5 -- op tracker -- seq: 46829, time: 2014-11-08 18:14:35.591382, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9dd5b16/ 1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8275> 2014-11-08 18:14:35.591976 7fc5a152e700 5 -- op tracker -- seq: 46829, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9dd5b16/ 1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8274> 2014-11-08 18:14:35.592190 7fc5afb15700 5 -- op tracker -- seq: 46829, time: 2014-11-08 18:14:35.592189, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9dd5b16/1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8273> 2014-11-08 18:14:35.592372 7fc5afb15700 5 -- op tracker -- seq: 46829, time: 2014-11-08 18:14:35.592372, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(253d5b16/10000019b04.000002ef/head//4, version: 397'1028, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(253d5b16/10000019b04.000002ef/head//4@397'1028, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(134d5b16/1000002b8f2.00000010/head//4, version: 3473'7123, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(134d5b16/1000002b8f2.00000010/head//4@3473'7123, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6cbd5b16/1000002b0e6.00000011/head//4, version: 4157'7999, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6cbd5b16/1000002b0e6.00000011/head//4@4157'7999, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9dd5b16/1000002af7c.000000b7/head//4, version: 4157'7799, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9dd5b16/ 1000002af7c.000000b7/head//4@4157'7799, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d0ed5b16/1000001b3fe.0000001f/head//4, version: 3033'2836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d0ed5b16/1000001b3fe.0000001f/head//4@3033'2836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8272> 2014-11-08 18:14:35.671351 7fc5c362f700 5 osd.9 5773 tick -8271> 2014-11-08 18:14:35.671399 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -8270> 2014-11-08 18:14:35.671409 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(6 pgs tid 1325 v 0) v1 -- ?+0 0x22d7a680 con 0x44ed6a0 -8269> 2014-11-08 18:14:35.743180 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) v1 -- ?+0 0x211d7580 con 0x42c86e0 -8268> 2014-11-08 18:14:35.743230 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d7580 con 0x42c86e0 -8267> 2014-11-08 18:14:35.743229 7fc5ac30e700 5 -- op tracker -- seq: 46819, time: 2014-11-08 18:14:35.743229, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4400, to_read={ca5df418/1000001b502.00000378/head//4=0,4194304,246df418/1000002bdbc.000000fe/head//4=0,4194304,611ef418/1000002a661.00000011/head//4=0,4194304,8a2ef418/10000019d91.00000026/head//4=0,4194304,318ef418/1000002a454.00000037/head//4=0,4194304}, attrs_to_read=)) -8266> 2014-11-08 18:14:35.743245 7fc5b9328700 5 -- op tracker -- seq: 46830, time: 2014-11-08 18:14:35.743216, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) -8265> 2014-11-08 18:14:35.743249 7fc5b9328700 5 -- op tracker -- seq: 46830, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) -8264> 2014-11-08 18:14:35.743251 7fc5b9328700 5 -- op tracker -- seq: 46830, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) -8263> 2014-11-08 18:14:35.743253 7fc5b9328700 5 -- op tracker -- seq: 46830, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) -8262> 2014-11-08 18:14:35.743280 7fc5ac30e700 5 -- op tracker -- seq: 46830, time: 2014-11-08 18:14:35.743280, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) -8261> 2014-11-08 18:14:35.743322 7fc5ac30e700 5 -- op tracker -- seq: 46830, time: 2014-11-08 18:14:35.743321, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=0)) -8260> 2014-11-08 18:14:35.743353 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x21223660 con 0x21468c60 -8259> 2014-11-08 18:14:35.743367 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(629bbddd/1000001a794.000001d6/head//4)]) v2 -- ?+0 0x1d85b800 con 0x21468c60 -8258> 2014-11-08 18:14:35.859831 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59234, attrs_read=0)) v1 -- ?+0 0x1c8bb200 con 0x21468c60 -8257> 2014-11-08 18:14:35.859881 7fc5af314700 5 -- op tracker -- seq: 46827, time: 2014-11-08 18:14:35.859881, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59234, to_read={827e314e/1000001b48b.0000004b/head//4=0,4194304}, attrs_to_read=)) -8256> 2014-11-08 18:14:35.913639 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3733 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:35.913118) v2 ==== 47+0+0 (4139767044 0 0) 0x2286ab20 con 0x1cce3700 -8255> 2014-11-08 18:14:35.913676 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:35.913118) v2 -- ?+0 0x2336d0a0 con 0x1cce3700 -8254> 2014-11-08 18:14:35.913907 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3733 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:35.913118) v2 ==== 47+0+0 (4139767044 0 0) 0x236832a0 con 0x1cce3180 -8253> 2014-11-08 18:14:35.913921 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:35.913118) v2 -- ?+0 0x1d043de0 con 0x1cce3180 -8252> 2014-11-08 18:14:35.926820 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1577 ==== pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) v3 ==== 757+0+0 (4200415157 0 0) 0x1e58e300 con 0x21468c60 -8251> 2014-11-08 18:14:35.926849 7fc5a39d2700 5 -- op tracker -- seq: 46831, time: 2014-11-08 18:14:35.926730, event: header_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8250> 2014-11-08 18:14:35.926857 7fc5a39d2700 5 -- op tracker -- seq: 46831, time: 2014-11-08 18:14:35.926735, event: throttled, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8249> 2014-11-08 18:14:35.926860 7fc5a39d2700 5 -- op tracker -- seq: 46831, time: 2014-11-08 18:14:35.926813, event: all_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8248> 2014-11-08 18:14:35.926863 7fc5a39d2700 5 -- op tracker -- seq: 46831, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8247> 2014-11-08 18:14:35.926919 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1578 ==== MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) v1 ==== 377+0+0 (43110500 0 0) 0x22878fc0 con 0x21468c60 -8246> 2014-11-08 18:14:35.926930 7fc5a39d2700 5 -- op tracker -- seq: 46832, time: 2014-11-08 18:14:35.926888, event: header_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) -8245> 2014-11-08 18:14:35.926935 7fc5a39d2700 5 -- op tracker -- seq: 46832, time: 2014-11-08 18:14:35.926889, event: throttled, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) -8244> 2014-11-08 18:14:35.926940 7fc5a39d2700 5 -- op tracker -- seq: 46832, time: 2014-11-08 18:14:35.926918, event: all_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) -8243> 2014-11-08 18:14:35.926945 7fc5a39d2700 5 -- op tracker -- seq: 46832, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) -8242> 2014-11-08 18:14:35.926983 7fc5acb0f700 5 -- op tracker -- seq: 46831, time: 2014-11-08 18:14:35.926982, event: reached_pg, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8241> 2014-11-08 18:14:35.926992 7fc5acb0f700 5 -- op tracker -- seq: 46831, time: 2014-11-08 18:14:35.926992, event: started, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8240> 2014-11-08 18:14:35.927045 7fc5acb0f700 5 -- op tracker -- seq: 46831, time: 2014-11-08 18:14:35.927045, event: done, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 7d27678f/1000001a107.0000002f/head//4) -8239> 2014-11-08 18:14:35.927059 7fc5acb0f700 5 -- op tracker -- seq: 46832, time: 2014-11-08 18:14:35.927059, event: reached_pg, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) -8238> 2014-11-08 18:14:35.989406 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1579 ==== pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) v3 ==== 757+0+0 (3581404571 0 0) 0x1e588900 con 0x21468c60 -8237> 2014-11-08 18:14:35.989447 7fc5a39d2700 5 -- op tracker -- seq: 46833, time: 2014-11-08 18:14:35.989296, event: header_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8236> 2014-11-08 18:14:35.989457 7fc5a39d2700 5 -- op tracker -- seq: 46833, time: 2014-11-08 18:14:35.989299, event: throttled, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8235> 2014-11-08 18:14:35.989460 7fc5a39d2700 5 -- op tracker -- seq: 46833, time: 2014-11-08 18:14:35.989398, event: all_read, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8234> 2014-11-08 18:14:35.989463 7fc5a39d2700 5 -- op tracker -- seq: 46833, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8233> 2014-11-08 18:14:35.989515 7fc5af314700 5 -- op tracker -- seq: 46833, time: 2014-11-08 18:14:35.989515, event: reached_pg, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8232> 2014-11-08 18:14:35.989527 7fc5af314700 5 -- op tracker -- seq: 46833, time: 2014-11-08 18:14:35.989526, event: started, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8231> 2014-11-08 18:14:35.989576 7fc5af314700 5 -- op tracker -- seq: 46833, time: 2014-11-08 18:14:35.989576, event: done, op: pg_backfill(progress 4.5ds0 e 5773/5773 lb 629bbddd/1000001a794.000001d6/head//4) -8230> 2014-11-08 18:14:36.022926 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8480 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) v1 ==== 10488228+0+0 (658870626 0 0) 0x224b8280 con 0x1c5c0580 -8229> 2014-11-08 18:14:36.022961 7fc5a152e700 5 -- op tracker -- seq: 46834, time: 2014-11-08 18:14:35.689455, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) -8228> 2014-11-08 18:14:36.022971 7fc5a152e700 5 -- op tracker -- seq: 46834, time: 2014-11-08 18:14:35.689460, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) -8227> 2014-11-08 18:14:36.022975 7fc5a152e700 5 -- op tracker -- seq: 46834, time: 2014-11-08 18:14:36.022916, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) -8226> 2014-11-08 18:14:36.022979 7fc5a152e700 5 -- op tracker -- seq: 46834, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) -8225> 2014-11-08 18:14:36.023015 7fc5ac30e700 5 -- op tracker -- seq: 46834, time: 2014-11-08 18:14:36.023015, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) -8224> 2014-11-08 18:14:36.046293 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(ca5df418/1000001b502.00000378/head//4, version: 3033'3278, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca5df418/1000001b502.00000378/head//4@3033'3278, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(246df418/1000002bdbc.000000fe/head//4, version: 4864'8342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(246df418/1000002bdbc.000000fe/head//4@4864'8342, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(611ef418/1000002a661.00000011/head//4, version: 3159'5265, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(611ef418/1000002a661.00000011/head//4@3159'5265, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a2ef418/10000019d91.00000026/head//4, version: 397'1547, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a2ef418/ 10000019d91.00000026/head//4@397'1547, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(318ef418/1000002a454.00000037/head//4, version: 3033'5033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(318ef418/1000002a454.00000037/head//4@3033'5033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3aba00 con 0x1e50aec0 -8223> 2014-11-08 18:14:36.052748 7fc5ac30e700 5 -- op tracker -- seq: 46834, time: 2014-11-08 18:14:36.052748, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4400, attrs_read=5)) -8222> 2014-11-08 18:14:36.084338 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) v1 -- ?+0 0x211d6e00 con 0x42c86e0 -8221> 2014-11-08 18:14:36.084366 7fc5ad310700 5 -- op tracker -- seq: 46828, time: 2014-11-08 18:14:36.084366, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4402, to_read={78ebd484/1000001a34e.00000036/head//4=0,4194304,586cd484/1000002bca6.0000008e/head//4=0,4194304,516dd484/10000029456.00000006/head//4=0,4194304}, attrs_to_read=)) -8220> 2014-11-08 18:14:36.084370 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d6e00 con 0x42c86e0 -8219> 2014-11-08 18:14:36.084381 7fc5b9328700 5 -- op tracker -- seq: 46835, time: 2014-11-08 18:14:36.084358, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) -8218> 2014-11-08 18:14:36.084385 7fc5b9328700 5 -- op tracker -- seq: 46835, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) -8217> 2014-11-08 18:14:36.084387 7fc5b9328700 5 -- op tracker -- seq: 46835, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) -8216> 2014-11-08 18:14:36.084389 7fc5b9328700 5 -- op tracker -- seq: 46835, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) -8215> 2014-11-08 18:14:36.084415 7fc5afb15700 5 -- op tracker -- seq: 46835, time: 2014-11-08 18:14:36.084415, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) -8214> 2014-11-08 18:14:36.084446 7fc5afb15700 5 -- op tracker -- seq: 46835, time: 2014-11-08 18:14:36.084446, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=0)) -8213> 2014-11-08 18:14:36.094174 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 491 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:36.091855) v2 ==== 47+0+0 (1954794943 0 0) 0x24886ea0 con 0x1eb9c8e0 -8212> 2014-11-08 18:14:36.094207 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:36.091855) v2 -- ?+0 0x2286ab20 con 0x1eb9c8e0 -8211> 2014-11-08 18:14:36.095239 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 491 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:36.091855) v2 ==== 47+0+0 (1954794943 0 0) 0x22e5e180 con 0x1eb9d800 -8210> 2014-11-08 18:14:36.095255 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:36.091855) v2 -- ?+0 0x236832a0 con 0x1eb9d800 -8209> 2014-11-08 18:14:36.155380 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) v1 -- ?+0 0x1c89bc00 con 0x42c86e0 -8208> 2014-11-08 18:14:36.155431 7fc5aeb13700 5 -- op tracker -- seq: 46824, time: 2014-11-08 18:14:36.155431, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4401, to_read={a658dd13/1000001a207.000005b8/head//4=0,4194304,fd19dd13/1000002a5c7.00000003/head//4=0,4194304,da99dd13/1000000e4b7.000003ed/head//4=0,4194304,88aadd13/10000029db8.0000000b/head//4=0,4194304}, attrs_to_read=)) -8207> 2014-11-08 18:14:36.155455 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x21224b00 con 0x1c5c0580 -8206> 2014-11-08 18:14:36.155473 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(1037fcb5/1000002965c.0000000e/head//4)]) v2 -- ?+0 0x1d5b2200 con 0x1c5c0580 -8205> 2014-11-08 18:14:36.155479 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1c89bc00 con 0x42c86e0 -8204> 2014-11-08 18:14:36.155490 7fc5b9328700 5 -- op tracker -- seq: 46836, time: 2014-11-08 18:14:36.155418, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) -8203> 2014-11-08 18:14:36.155494 7fc5b9328700 5 -- op tracker -- seq: 46836, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) -8202> 2014-11-08 18:14:36.155496 7fc5b9328700 5 -- op tracker -- seq: 46836, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) -8201> 2014-11-08 18:14:36.155498 7fc5b9328700 5 -- op tracker -- seq: 46836, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) -8200> 2014-11-08 18:14:36.155536 7fc5ac30e700 5 -- op tracker -- seq: 46836, time: 2014-11-08 18:14:36.155536, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) -8199> 2014-11-08 18:14:36.155632 7fc5ac30e700 5 -- op tracker -- seq: 46836, time: 2014-11-08 18:14:36.155632, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=0)) -8198> 2014-11-08 18:14:36.165119 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8481 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) v1 ==== 6292958+0+0 (2860256279 0 0) 0x44a5a00 con 0x1c5c0580 -8197> 2014-11-08 18:14:36.165191 7fc5a152e700 5 -- op tracker -- seq: 46837, time: 2014-11-08 18:14:36.023015, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) -8196> 2014-11-08 18:14:36.165201 7fc5a152e700 5 -- op tracker -- seq: 46837, time: 2014-11-08 18:14:36.023016, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) -8195> 2014-11-08 18:14:36.165205 7fc5a152e700 5 -- op tracker -- seq: 46837, time: 2014-11-08 18:14:36.165109, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) -8194> 2014-11-08 18:14:36.165209 7fc5a152e700 5 -- op tracker -- seq: 46837, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) -8193> 2014-11-08 18:14:36.165246 7fc5ad310700 5 -- op tracker -- seq: 46837, time: 2014-11-08 18:14:36.165246, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) -8192> 2014-11-08 18:14:36.165284 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8482 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) v3 ==== 757+0+0 (201468196 0 0) 0x1d652d00 con 0x1c5c0580 -8191> 2014-11-08 18:14:36.165293 7fc5a152e700 5 -- op tracker -- seq: 46838, time: 2014-11-08 18:14:36.165248, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8190> 2014-11-08 18:14:36.165297 7fc5a152e700 5 -- op tracker -- seq: 46838, time: 2014-11-08 18:14:36.165250, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8189> 2014-11-08 18:14:36.165300 7fc5a152e700 5 -- op tracker -- seq: 46838, time: 2014-11-08 18:14:36.165281, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8188> 2014-11-08 18:14:36.165303 7fc5a152e700 5 -- op tracker -- seq: 46838, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8187> 2014-11-08 18:14:36.165324 7fc5adb11700 5 -- op tracker -- seq: 46838, time: 2014-11-08 18:14:36.165324, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8186> 2014-11-08 18:14:36.165331 7fc5adb11700 5 -- op tracker -- seq: 46838, time: 2014-11-08 18:14:36.165331, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8185> 2014-11-08 18:14:36.165388 7fc5a152e700 5 -- op tracker -- seq: 46838, time: 2014-11-08 18:14:36.165388, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1037fcb5/1000002965c.0000000e/head//4) -8184> 2014-11-08 18:14:36.181568 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(78ebd484/1000001a34e.00000036/head//4, version: 397'1863, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(78ebd484/1000001a34e.00000036/head//4@397'1863, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(586cd484/1000002bca6.0000008e/head//4, version: 3485'7456, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(586cd484/1000002bca6.0000008e/head//4@3485'7456, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(516dd484/10000029456.00000006/head//4, version: 3033'3963, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(516dd484/10000029456.00000006/head//4@3033'3963, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d5b4c00 con 0x1e50aec0 -8183> 2014-11-08 18:14:36.181696 7fc5ad310700 5 -- op tracker -- seq: 46837, time: 2014-11-08 18:14:36.181696, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4402, attrs_read=3)) -8182> 2014-11-08 18:14:36.202730 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1580 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (374649009 0 0) 0x22878d80 con 0x21468c60 -8181> 2014-11-08 18:14:36.202768 7fc5a39d2700 5 -- op tracker -- seq: 46839, time: 2014-11-08 18:14:36.202624, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) -8180> 2014-11-08 18:14:36.202778 7fc5a39d2700 5 -- op tracker -- seq: 46839, time: 2014-11-08 18:14:36.202629, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) -8179> 2014-11-08 18:14:36.202781 7fc5a39d2700 5 -- op tracker -- seq: 46839, time: 2014-11-08 18:14:36.202721, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) -8178> 2014-11-08 18:14:36.202784 7fc5a39d2700 5 -- op tracker -- seq: 46839, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) -8177> 2014-11-08 18:14:36.202866 7fc5af314700 5 -- op tracker -- seq: 46839, time: 2014-11-08 18:14:36.202866, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) -8176> 2014-11-08 18:14:36.260477 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59238, attrs_read=0)) v1 -- ?+0 0x1c89e400 con 0x21468c60 -8175> 2014-11-08 18:14:36.260526 7fc5af314700 5 -- op tracker -- seq: 46839, time: 2014-11-08 18:14:36.260525, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59238, to_read={380f314e/1000002a40a.00000019/head//4=0,4194304}, attrs_to_read=)) -8174> 2014-11-08 18:14:36.281105 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2186 ==== MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 4196083+0+0 (1485450961 0 0) 0x1ebb5c00 con 0x1e50aec0 -8173> 2014-11-08 18:14:36.281157 7fc5a56ef700 5 -- op tracker -- seq: 46840, time: 2014-11-08 18:14:36.101702, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8172> 2014-11-08 18:14:36.281176 7fc5a56ef700 5 -- op tracker -- seq: 46840, time: 2014-11-08 18:14:36.101706, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8171> 2014-11-08 18:14:36.281186 7fc5a56ef700 5 -- op tracker -- seq: 46840, time: 2014-11-08 18:14:36.281100, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8170> 2014-11-08 18:14:36.281197 7fc5a56ef700 5 -- op tracker -- seq: 46840, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8169> 2014-11-08 18:14:36.281244 7fc5af314700 5 -- op tracker -- seq: 46840, time: 2014-11-08 18:14:36.281244, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8168> 2014-11-08 18:14:36.281368 7fc5af314700 5 -- op tracker -- seq: 46840, time: 2014-11-08 18:14:36.281368, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(a06b90b7/10000019b73.0000002a/head//4, version: 397'1207, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a06b90b7/10000019b73.0000002a/head//4@397'1207, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(dfcb90b7/1000001b501.00000ad0/head//4, version: 3033'3200, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(dfcb90b7/1000001b501.00000ad0/head//4@3033'3200, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8167> 2014-11-08 18:14:36.329714 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x21223c00 con 0x1c5c0580 -8166> 2014-11-08 18:14:36.329760 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x23684740 con 0x1c5c0580 -8165> 2014-11-08 18:14:36.329778 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d5e1a40 con 0x1c5c0580 -8164> 2014-11-08 18:14:36.329802 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25e84380 con 0x1c5c0580 -8163> 2014-11-08 18:14:36.329809 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22e9b840 con 0x1c5c0580 -8162> 2014-11-08 18:14:36.448167 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8483 ==== MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2910524485 0 0) 0x1d5b2200 con 0x1c5c0580 -8161> 2014-11-08 18:14:36.448216 7fc5a152e700 5 -- op tracker -- seq: 46841, time: 2014-11-08 18:14:36.358418, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8160> 2014-11-08 18:14:36.448234 7fc5a152e700 5 -- op tracker -- seq: 46841, time: 2014-11-08 18:14:36.358421, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8159> 2014-11-08 18:14:36.448249 7fc5a152e700 5 -- op tracker -- seq: 46841, time: 2014-11-08 18:14:36.448104, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8158> 2014-11-08 18:14:36.448261 7fc5a152e700 5 -- op tracker -- seq: 46841, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8157> 2014-11-08 18:14:36.448310 7fc5b0316700 5 -- op tracker -- seq: 46841, time: 2014-11-08 18:14:36.448310, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8156> 2014-11-08 18:14:36.448529 7fc5b0316700 5 -- op tracker -- seq: 46841, time: 2014-11-08 18:14:36.448529, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(1687fcb5/1000002b03d.00000046/head//4, version: 4157'7753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1687fcb5/1000002b03d.00000046/head//4@4157'7753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8155> 2014-11-08 18:14:36.492153 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 313 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:36.489243) v2 ==== 47+0+0 (2701508962 0 0) 0x2026a580 con 0x1eb9c780 -8154> 2014-11-08 18:14:36.492167 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 313 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:36.489243) v2 ==== 47+0+0 (2701508962 0 0) 0x257c7bc0 con 0x1eb9d6a0 -8153> 2014-11-08 18:14:36.492207 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:36.489243) v2 -- ?+0 0x24886ea0 con 0x1eb9c780 -8152> 2014-11-08 18:14:36.492241 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:36.489243) v2 -- ?+0 0x22e5e180 con 0x1eb9d6a0 -8151> 2014-11-08 18:14:36.505321 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.8fs0 5773 ECSubReadReply(tid=59236, attrs_read=0)) v1 -- ?+0 0x21a48780 con 0x21468c60 -8150> 2014-11-08 18:14:36.505363 7fc5acb0f700 5 -- op tracker -- seq: 46832, time: 2014-11-08 18:14:36.505363, event: done, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59236, to_read={12e8678f/1000002af7f.0000001b/head//4=0,4194304,5439678f/1000001a8e7.0000001c/head//4=0,4194304,eb79678f/1000002bbec.00000004/head//4=0,4194304,76aa678f/1000002a47e.00000015/head//4=0,4194304}, attrs_to_read=)) -8149> 2014-11-08 18:14:36.517549 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x22e9b0c0 con 0x1e50aec0 -8148> 2014-11-08 18:14:36.517595 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x22e99680 con 0x1e50aec0 -8147> 2014-11-08 18:14:36.568242 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1500 ==== pg_stats_ack(6 pgs tid 1325) v1 ==== 178+0+0 (4159406308 0 0) 0x1f4c0d20 con 0x44ed6a0 -8146> 2014-11-08 18:14:36.593619 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22e9eea0 con 0x1c5c0580 -8145> 2014-11-08 18:14:36.659034 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2187 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) v1 ==== 8390589+0+0 (795807178 0 0) 0x44a6e00 con 0x1e50aec0 -8144> 2014-11-08 18:14:36.659116 7fc5a56ef700 5 -- op tracker -- seq: 46842, time: 2014-11-08 18:14:36.281243, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) -8143> 2014-11-08 18:14:36.659128 7fc5a56ef700 5 -- op tracker -- seq: 46842, time: 2014-11-08 18:14:36.281244, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) -8142> 2014-11-08 18:14:36.659137 7fc5a56ef700 5 -- op tracker -- seq: 46842, time: 2014-11-08 18:14:36.659027, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) -8141> 2014-11-08 18:14:36.659146 7fc5a56ef700 5 -- op tracker -- seq: 46842, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) -8140> 2014-11-08 18:14:36.659176 7fc5aeb13700 5 -- op tracker -- seq: 46842, time: 2014-11-08 18:14:36.659176, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) -8139> 2014-11-08 18:14:36.671503 7fc5c362f700 5 osd.9 5773 tick -8138> 2014-11-08 18:14:36.694993 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2188 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) v2 ==== 110+0+0 (608106555 0 0) 0x1d3aba00 con 0x1e50aec0 -8137> 2014-11-08 18:14:36.695036 7fc5a56ef700 5 -- op tracker -- seq: 46843, time: 2014-11-08 18:14:36.694894, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) -8136> 2014-11-08 18:14:36.695051 7fc5a56ef700 5 -- op tracker -- seq: 46843, time: 2014-11-08 18:14:36.694898, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) -8135> 2014-11-08 18:14:36.695060 7fc5a56ef700 5 -- op tracker -- seq: 46843, time: 2014-11-08 18:14:36.694969, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) -8134> 2014-11-08 18:14:36.695070 7fc5a56ef700 5 -- op tracker -- seq: 46843, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) -8133> 2014-11-08 18:14:36.695656 7fc5adb11700 5 -- op tracker -- seq: 46843, time: 2014-11-08 18:14:36.695656, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) -8132> 2014-11-08 18:14:36.695744 7fc5adb11700 5 -- op tracker -- seq: 46843, time: 2014-11-08 18:14:36.695744, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(adfc1d83/1000002a6c6.00000008/head//4)]) -8131> 2014-11-08 18:14:36.703187 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2189 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22e99680 con 0x1e50aec0 -8130> 2014-11-08 18:14:36.703220 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703061, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8129> 2014-11-08 18:14:36.703229 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703065, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8128> 2014-11-08 18:14:36.703232 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703133, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8127> 2014-11-08 18:14:36.703234 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703215, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8126> 2014-11-08 18:14:36.703237 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703237, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8125> 2014-11-08 18:14:36.703246 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703246, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8124> 2014-11-08 18:14:36.703251 7fc5b9b29700 5 -- op tracker -- seq: 46844, time: 2014-11-08 18:14:36.703251, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8123> 2014-11-08 18:14:36.703318 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2190 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22e9b0c0 con 0x1e50aec0 -8122> 2014-11-08 18:14:36.703328 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703286, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8121> 2014-11-08 18:14:36.703331 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703287, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8120> 2014-11-08 18:14:36.703334 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703308, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8119> 2014-11-08 18:14:36.703336 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703327, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8118> 2014-11-08 18:14:36.703339 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703339, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8117> 2014-11-08 18:14:36.703343 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703343, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8116> 2014-11-08 18:14:36.703346 7fc5b9b29700 5 -- op tracker -- seq: 46845, time: 2014-11-08 18:14:36.703346, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8115> 2014-11-08 18:14:36.703757 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2191 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1d1b6ae0 con 0x1e50aec0 -8114> 2014-11-08 18:14:36.703768 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703367, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8113> 2014-11-08 18:14:36.703771 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703367, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8112> 2014-11-08 18:14:36.703774 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703388, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8111> 2014-11-08 18:14:36.703776 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703766, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8110> 2014-11-08 18:14:36.703779 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703779, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8109> 2014-11-08 18:14:36.703782 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703782, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8108> 2014-11-08 18:14:36.703786 7fc5b9b29700 5 -- op tracker -- seq: 46846, time: 2014-11-08 18:14:36.703786, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8107> 2014-11-08 18:14:36.703838 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2192 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x23d76540 con 0x1e50aec0 -8106> 2014-11-08 18:14:36.703848 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703807, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8105> 2014-11-08 18:14:36.703852 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703807, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8104> 2014-11-08 18:14:36.703855 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703828, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8103> 2014-11-08 18:14:36.703857 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703847, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8102> 2014-11-08 18:14:36.703860 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703860, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8101> 2014-11-08 18:14:36.703863 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703863, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8100> 2014-11-08 18:14:36.703867 7fc5b9b29700 5 -- op tracker -- seq: 46847, time: 2014-11-08 18:14:36.703867, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8099> 2014-11-08 18:14:36.703924 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2193 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x23d701e0 con 0x1e50aec0 -8098> 2014-11-08 18:14:36.703934 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703886, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -8097> 2014-11-08 18:14:36.703937 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703886, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -8096> 2014-11-08 18:14:36.703940 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703914, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -8095> 2014-11-08 18:14:36.703942 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703933, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -8094> 2014-11-08 18:14:36.703945 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703945, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -8093> 2014-11-08 18:14:36.703949 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703949, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -8092> 2014-11-08 18:14:36.703952 7fc5b9b29700 5 -- op tracker -- seq: 46848, time: 2014-11-08 18:14:36.703952, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -8091> 2014-11-08 18:14:36.711695 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=8c4e1d83/1000001f006.00000000/head//4)) v1 -- ?+0 0x224ac140 con 0x1c5c0580 -8090> 2014-11-08 18:14:36.711735 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x224ac5c0 con 0x42c86e0 -8089> 2014-11-08 18:14:36.711746 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is cd9e1d83/1000001a9f6.00000159/head//4 -8088> 2014-11-08 18:14:36.711773 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb adfc1d83/1000002a6c6.00000008/head//4) v3 -- ?+0 0x1d653600 con 0x1e50aec0 -8087> 2014-11-08 18:14:36.716723 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x224ac5c0 con 0x42c86e0 -8086> 2014-11-08 18:14:36.716764 7fc5b9328700 5 -- op tracker -- seq: 46849, time: 2014-11-08 18:14:36.711741, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) -8085> 2014-11-08 18:14:36.716772 7fc5b9328700 5 -- op tracker -- seq: 46849, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) -8084> 2014-11-08 18:14:36.716775 7fc5b9328700 5 -- op tracker -- seq: 46849, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) -8083> 2014-11-08 18:14:36.716778 7fc5b9328700 5 -- op tracker -- seq: 46849, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) -8082> 2014-11-08 18:14:36.716923 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1581 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (1379440962 0 0) 0x22878b40 con 0x21468c60 -8081> 2014-11-08 18:14:36.716934 7fc5a39d2700 5 -- op tracker -- seq: 46850, time: 2014-11-08 18:14:36.716848, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) -8080> 2014-11-08 18:14:36.716938 7fc5a39d2700 5 -- op tracker -- seq: 46850, time: 2014-11-08 18:14:36.716851, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) -8079> 2014-11-08 18:14:36.716941 7fc5a39d2700 5 -- op tracker -- seq: 46850, time: 2014-11-08 18:14:36.716917, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) -8078> 2014-11-08 18:14:36.716944 7fc5a39d2700 5 -- op tracker -- seq: 46850, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) -8077> 2014-11-08 18:14:36.716973 7fc5b0316700 5 -- op tracker -- seq: 46849, time: 2014-11-08 18:14:36.716973, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) -8076> 2014-11-08 18:14:36.717109 7fc5af314700 5 -- op tracker -- seq: 46850, time: 2014-11-08 18:14:36.717109, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) -8075> 2014-11-08 18:14:36.721551 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(a658dd13/1000001a207.000005b8/head//4, version: 397'1823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a658dd13/1000001a207.000005b8/head//4@397'1823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fd19dd13/1000002a5c7.00000003/head//4, version: 3033'4905, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd19dd13/1000002a5c7.00000003/head//4@3033'4905, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da99dd13/1000000e4b7.000003ed/head//4, version: 397'246, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da99dd13/1000000e4b7.000003ed/head//4@397'246, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88aadd13/10000029db8.0000000b/head//4, version: 3033'4451, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88aadd13/10000029db8.0000000b/head// 4@3033'4451, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d85e400 con 0x1c5c0580 -8074> 2014-11-08 18:14:36.721618 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(a658dd13/1000001a207.000005b8/head//4, version: 397'1823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a658dd13/1000001a207.000005b8/head//4@397'1823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fd19dd13/1000002a5c7.00000003/head//4, version: 3033'4905, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd19dd13/1000002a5c7.00000003/head//4@3033'4905, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da99dd13/1000000e4b7.000003ed/head//4, version: 397'246, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da99dd13/1000000e4b7.000003ed/head//4@397'246, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88aadd13/10000029db8.0000000b/head//4, version: 3033'4451, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88aadd13/10000029db8.0000000b/head// 4@3033'4451, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d85c200 con 0x1e50aec0 -8073> 2014-11-08 18:14:36.721681 7fc5aeb13700 5 -- op tracker -- seq: 46842, time: 2014-11-08 18:14:36.721681, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4401, attrs_read=4)) -8072> 2014-11-08 18:14:36.837188 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59240, attrs_read=0)) v1 -- ?+0 0x1c898f00 con 0x21468c60 -8071> 2014-11-08 18:14:36.837396 7fc5af314700 5 -- op tracker -- seq: 46850, time: 2014-11-08 18:14:36.837396, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59240, to_read={fa9f314e/1000002aba8.00000227/head//4=0,4194304}, attrs_to_read=)) -8070> 2014-11-08 18:14:36.852590 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) v1 -- ?+0 0x224beb80 con 0x42c86e0 -8069> 2014-11-08 18:14:36.852642 7fc5b0316700 5 -- op tracker -- seq: 46849, time: 2014-11-08 18:14:36.852641, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4403, to_read={8c4e1d83/1000001f006.00000000/head//4=0,4194304}, attrs_to_read=)) -8068> 2014-11-08 18:14:36.852710 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224beb80 con 0x42c86e0 -8067> 2014-11-08 18:14:36.852724 7fc5b9328700 5 -- op tracker -- seq: 46851, time: 2014-11-08 18:14:36.852630, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) -8066> 2014-11-08 18:14:36.852728 7fc5b9328700 5 -- op tracker -- seq: 46851, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) -8065> 2014-11-08 18:14:36.852730 7fc5b9328700 5 -- op tracker -- seq: 46851, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) -8064> 2014-11-08 18:14:36.852732 7fc5b9328700 5 -- op tracker -- seq: 46851, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) -8063> 2014-11-08 18:14:36.852770 7fc5adb11700 5 -- op tracker -- seq: 46851, time: 2014-11-08 18:14:36.852770, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) -8062> 2014-11-08 18:14:36.852817 7fc5adb11700 5 -- op tracker -- seq: 46851, time: 2014-11-08 18:14:36.852817, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=0)) -8061> 2014-11-08 18:14:36.856285 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8484 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) v1 ==== 39818+0+0 (3603728064 0 0) 0x44a1900 con 0x1c5c0580 -8060> 2014-11-08 18:14:36.856317 7fc5a152e700 5 -- op tracker -- seq: 46852, time: 2014-11-08 18:14:36.854543, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) -8059> 2014-11-08 18:14:36.856326 7fc5a152e700 5 -- op tracker -- seq: 46852, time: 2014-11-08 18:14:36.854548, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) -8058> 2014-11-08 18:14:36.856329 7fc5a152e700 5 -- op tracker -- seq: 46852, time: 2014-11-08 18:14:36.856282, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) -8057> 2014-11-08 18:14:36.856333 7fc5a152e700 5 -- op tracker -- seq: 46852, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) -8056> 2014-11-08 18:14:36.856418 7fc5b0316700 5 -- op tracker -- seq: 46852, time: 2014-11-08 18:14:36.856418, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) -8055> 2014-11-08 18:14:36.856571 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(8c4e1d83/1000001f006.00000000/head//4, version: 3033'3458, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(8c4e1d83/1000001f006.00000000/head//4@3033'3458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d5b2200 con 0x1e50aec0 -8054> 2014-11-08 18:14:36.856607 7fc5b0316700 5 -- op tracker -- seq: 46852, time: 2014-11-08 18:14:36.856607, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4403, attrs_read=1)) -8053> 2014-11-08 18:14:36.859741 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2194 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x23d76cc0 con 0x1e50aec0 -8052> 2014-11-08 18:14:36.859759 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859655, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -8051> 2014-11-08 18:14:36.859763 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859657, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -8050> 2014-11-08 18:14:36.859766 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859723, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -8049> 2014-11-08 18:14:36.859769 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859755, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -8048> 2014-11-08 18:14:36.859772 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859772, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -8047> 2014-11-08 18:14:36.859779 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859779, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -8046> 2014-11-08 18:14:36.859784 7fc5b9b29700 5 -- op tracker -- seq: 46853, time: 2014-11-08 18:14:36.859784, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -8045> 2014-11-08 18:14:36.860201 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2195 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x23d70960 con 0x1e50aec0 -8044> 2014-11-08 18:14:36.860215 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860164, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -8043> 2014-11-08 18:14:36.860219 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860165, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -8042> 2014-11-08 18:14:36.860222 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860190, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -8041> 2014-11-08 18:14:36.860224 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860213, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -8040> 2014-11-08 18:14:36.860229 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860229, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -8039> 2014-11-08 18:14:36.860295 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860295, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -8038> 2014-11-08 18:14:36.860300 7fc5b9b29700 5 -- op tracker -- seq: 46854, time: 2014-11-08 18:14:36.860300, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -8037> 2014-11-08 18:14:36.860308 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2196 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x23d76720 con 0x1e50aec0 -8036> 2014-11-08 18:14:36.860316 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860198, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -8035> 2014-11-08 18:14:36.860319 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860198, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -8034> 2014-11-08 18:14:36.860323 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860217, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -8033> 2014-11-08 18:14:36.860325 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860315, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -8032> 2014-11-08 18:14:36.860329 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860329, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -8031> 2014-11-08 18:14:36.860333 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860333, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -8030> 2014-11-08 18:14:36.860338 7fc5b9b29700 5 -- op tracker -- seq: 46855, time: 2014-11-08 18:14:36.860337, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -8029> 2014-11-08 18:14:36.965819 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1582 ==== MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/ 1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8392121+0+0 (2846208296 0 0) 0x1d85b800 con 0x21468c60 -8028> 2014-11-08 18:14:36.965879 7fc5a39d2700 5 -- op tracker -- seq: 46856, time: 2014-11-08 18:14:36.811315, event: header_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8027> 2014-11-08 18:14:36.965905 7fc5a39d2700 5 -- op tracker -- seq: 46856, time: 2014-11-08 18:14:36.811321, event: throttled, op: MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8026> 2014-11-08 18:14:36.965927 7fc5a39d2700 5 -- op tracker -- seq: 46856, time: 2014-11-08 18:14:36.965808, event: all_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8025> 2014-11-08 18:14:36.965948 7fc5a39d2700 5 -- op tracker -- seq: 46856, time: 0.000000, event: dispatched, op: MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/ 1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8024> 2014-11-08 18:14:36.966002 7fc5acb0f700 5 -- op tracker -- seq: 46856, time: 2014-11-08 18:14:36.966002, event: reached_pg, op: MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8023> 2014-11-08 18:14:36.966227 7fc5a39d2700 5 -- op tracker -- seq: 46856, time: 2014-11-08 18:14:36.966227, event: done, op: MOSDPGPush(4.8fs2 5773 [PushOp(12e8678f/1000002af7f.0000001b/head//4, version: 4157'7553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(12e8678f/1000002af7f.0000001b/head//4@4157'7553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5439678f/1000001a8e7.0000001c/head//4, version: 2829'2143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5439678f/1000001a8e7.0000001c/head//4@2829'2143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eb79678f/1000002bbec.00000004/head//4, version: 3478'7301, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eb79678f/1000002bbec.00000004/head//4@3478'7301, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(76aa678f/1000002a47e.00000015/head//4, version: 3033'4795, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(76aa678f/ 1000002a47e.00000015/head//4@3033'4795, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8022> 2014-11-08 18:14:37.105657 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1583 ==== MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (4147095196 0 0) 0x1d85b800 con 0x21468c60 -8021> 2014-11-08 18:14:37.112966 7fc5a39d2700 5 -- op tracker -- seq: 46857, time: 2014-11-08 18:14:37.002472, event: header_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8020> 2014-11-08 18:14:37.112990 7fc5a39d2700 5 -- op tracker -- seq: 46857, time: 2014-11-08 18:14:37.002475, event: throttled, op: MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8019> 2014-11-08 18:14:37.112999 7fc5a39d2700 5 -- op tracker -- seq: 46857, time: 2014-11-08 18:14:37.105644, event: all_read, op: MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8018> 2014-11-08 18:14:37.113007 7fc5a39d2700 5 -- op tracker -- seq: 46857, time: 0.000000, event: dispatched, op: MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8017> 2014-11-08 18:14:37.113052 7fc5af314700 5 -- op tracker -- seq: 46857, time: 2014-11-08 18:14:37.113052, event: reached_pg, op: MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8016> 2014-11-08 18:14:37.113200 7fc5af314700 5 -- op tracker -- seq: 46857, time: 2014-11-08 18:14:37.113200, event: done, op: MOSDPGPush(4.5ds0 5773 [PushOp(4bcbbddd/1000002a94c.0000009e/head//4, version: 3458'11019, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4bcbbddd/1000002a94c.0000009e/head//4@3458'11019, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -8015> 2014-11-08 18:14:37.204660 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22e9f9e0 con 0x21468c60 -8014> 2014-11-08 18:14:37.204782 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22e9c560 con 0x21468c60 -8013> 2014-11-08 18:14:37.204812 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x1eae1e00 con 0x21468c60 -8012> 2014-11-08 18:14:37.204822 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x1eae5280 con 0x21468c60 -8011> 2014-11-08 18:14:37.275426 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.5ds1 to 5734'16451 e5773) v1 -- ?+0 0x1eae10e0 con 0x21468c60 -8010> 2014-11-08 18:14:37.280163 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2197 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x23d74ce0 con 0x1e50aec0 -8009> 2014-11-08 18:14:37.280183 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280086, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -8008> 2014-11-08 18:14:37.280192 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280090, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -8007> 2014-11-08 18:14:37.280195 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280142, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -8006> 2014-11-08 18:14:37.280199 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280177, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -8005> 2014-11-08 18:14:37.280202 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280202, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -8004> 2014-11-08 18:14:37.280211 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280211, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -8003> 2014-11-08 18:14:37.280219 7fc5b9b29700 5 -- op tracker -- seq: 46858, time: 2014-11-08 18:14:37.280219, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -8002> 2014-11-08 18:14:37.280636 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2198 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x23d723a0 con 0x1e50aec0 -8001> 2014-11-08 18:14:37.280647 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280578, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -8000> 2014-11-08 18:14:37.280653 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280579, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7999> 2014-11-08 18:14:37.280656 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280614, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7998> 2014-11-08 18:14:37.280659 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280646, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7997> 2014-11-08 18:14:37.280663 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280663, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7996> 2014-11-08 18:14:37.280669 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280669, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7995> 2014-11-08 18:14:37.280674 7fc5b9b29700 5 -- op tracker -- seq: 46859, time: 2014-11-08 18:14:37.280674, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7994> 2014-11-08 18:14:37.280749 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2199 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x213bc740 con 0x1e50aec0 -7993> 2014-11-08 18:14:37.280760 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280697, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -7992> 2014-11-08 18:14:37.280763 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280698, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7991> 2014-11-08 18:14:37.280766 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280719, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7990> 2014-11-08 18:14:37.280768 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280758, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7989> 2014-11-08 18:14:37.280771 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280771, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7988> 2014-11-08 18:14:37.280775 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280775, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7987> 2014-11-08 18:14:37.280779 7fc5b9b29700 5 -- op tracker -- seq: 46860, time: 2014-11-08 18:14:37.280779, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7986> 2014-11-08 18:14:37.280785 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2200 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x213bad00 con 0x1e50aec0 -7985> 2014-11-08 18:14:37.280791 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280747, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -7984> 2014-11-08 18:14:37.280794 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280747, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7983> 2014-11-08 18:14:37.280796 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280766, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7982> 2014-11-08 18:14:37.280798 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280790, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7981> 2014-11-08 18:14:37.280801 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280801, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7980> 2014-11-08 18:14:37.280834 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280834, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7979> 2014-11-08 18:14:37.280838 7fc5b9b29700 5 -- op tracker -- seq: 46861, time: 2014-11-08 18:14:37.280838, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7978> 2014-11-08 18:14:37.294954 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1584 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (735271681 0 0) 0x22878900 con 0x21468c60 -7977> 2014-11-08 18:14:37.294984 7fc5a39d2700 5 -- op tracker -- seq: 46862, time: 2014-11-08 18:14:37.294875, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) -7976> 2014-11-08 18:14:37.294993 7fc5a39d2700 5 -- op tracker -- seq: 46862, time: 2014-11-08 18:14:37.294877, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) -7975> 2014-11-08 18:14:37.294997 7fc5a39d2700 5 -- op tracker -- seq: 46862, time: 2014-11-08 18:14:37.294948, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) -7974> 2014-11-08 18:14:37.295002 7fc5a39d2700 5 -- op tracker -- seq: 46862, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) -7973> 2014-11-08 18:14:37.295041 7fc5acb0f700 5 -- op tracker -- seq: 46862, time: 2014-11-08 18:14:37.295041, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) -7972> 2014-11-08 18:14:37.362384 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2201 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x213bb660 con 0x1e50aec0 -7971> 2014-11-08 18:14:37.362421 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362253, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -7970> 2014-11-08 18:14:37.362430 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362258, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -7969> 2014-11-08 18:14:37.362432 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362357, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -7968> 2014-11-08 18:14:37.362435 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362416, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -7967> 2014-11-08 18:14:37.362438 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362438, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -7966> 2014-11-08 18:14:37.362446 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362446, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -7965> 2014-11-08 18:14:37.362452 7fc5b9b29700 5 -- op tracker -- seq: 46863, time: 2014-11-08 18:14:37.362452, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -7964> 2014-11-08 18:14:37.405089 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8485 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x22e9eea0 con 0x1c5c0580 -7963> 2014-11-08 18:14:37.405128 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.404914, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -7962> 2014-11-08 18:14:37.405136 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.404918, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7961> 2014-11-08 18:14:37.405139 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.405053, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7960> 2014-11-08 18:14:37.405141 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.405123, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7959> 2014-11-08 18:14:37.405144 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.405144, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7958> 2014-11-08 18:14:37.405153 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.405152, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7957> 2014-11-08 18:14:37.405159 7fc5b9b29700 5 -- op tracker -- seq: 46864, time: 2014-11-08 18:14:37.405158, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7956> 2014-11-08 18:14:37.405169 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8486 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x22e9b840 con 0x1c5c0580 -7955> 2014-11-08 18:14:37.405175 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405094, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -7954> 2014-11-08 18:14:37.405178 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405095, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7953> 2014-11-08 18:14:37.405181 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405115, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7952> 2014-11-08 18:14:37.405183 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405174, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7951> 2014-11-08 18:14:37.405186 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405186, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7950> 2014-11-08 18:14:37.405189 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405189, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7949> 2014-11-08 18:14:37.405192 7fc5b9b29700 5 -- op tracker -- seq: 46865, time: 2014-11-08 18:14:37.405192, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7948> 2014-11-08 18:14:37.405197 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8487 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x25e84380 con 0x1c5c0580 -7947> 2014-11-08 18:14:37.405203 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405128, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -7946> 2014-11-08 18:14:37.405206 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405129, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7945> 2014-11-08 18:14:37.405208 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405147, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7944> 2014-11-08 18:14:37.405210 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405202, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7943> 2014-11-08 18:14:37.405213 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405213, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7942> 2014-11-08 18:14:37.405216 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405216, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7941> 2014-11-08 18:14:37.405219 7fc5b9b29700 5 -- op tracker -- seq: 46866, time: 2014-11-08 18:14:37.405219, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7940> 2014-11-08 18:14:37.405224 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8488 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1d5e1a40 con 0x1c5c0580 -7939> 2014-11-08 18:14:37.405230 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405159, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -7938> 2014-11-08 18:14:37.405233 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405160, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -7937> 2014-11-08 18:14:37.405235 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405184, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -7936> 2014-11-08 18:14:37.405237 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405229, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -7935> 2014-11-08 18:14:37.405241 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405241, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -7934> 2014-11-08 18:14:37.405245 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405245, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -7933> 2014-11-08 18:14:37.405249 7fc5b9b29700 5 -- op tracker -- seq: 46867, time: 2014-11-08 18:14:37.405249, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -7932> 2014-11-08 18:14:37.671648 7fc5c362f700 5 osd.9 5773 tick -7931> 2014-11-08 18:14:38.131123 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59241, attrs_read=0)) v1 -- ?+0 0x21a4c380 con 0x21468c60 -7930> 2014-11-08 18:14:38.131188 7fc5acb0f700 5 -- op tracker -- seq: 46862, time: 2014-11-08 18:14:38.131188, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59241, to_read={1baf314e/1000001b4ed.0000004b/head//4=0,4194304}, attrs_to_read=)) -7929> 2014-11-08 18:14:38.315023 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1585 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2769260814 0 0) 0x228786c0 con 0x21468c60 -7928> 2014-11-08 18:14:38.315055 7fc5a39d2700 5 -- op tracker -- seq: 46868, time: 2014-11-08 18:14:38.314932, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) -7927> 2014-11-08 18:14:38.315065 7fc5a39d2700 5 -- op tracker -- seq: 46868, time: 2014-11-08 18:14:38.314935, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) -7926> 2014-11-08 18:14:38.315069 7fc5a39d2700 5 -- op tracker -- seq: 46868, time: 2014-11-08 18:14:38.315017, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) -7925> 2014-11-08 18:14:38.315074 7fc5a39d2700 5 -- op tracker -- seq: 46868, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) -7924> 2014-11-08 18:14:38.315111 7fc5af314700 5 -- op tracker -- seq: 46868, time: 2014-11-08 18:14:38.315111, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) -7923> 2014-11-08 18:14:38.347607 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8489 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) v2 ==== 305+0+0 (464172493 0 0) 0x1d85e400 con 0x1c5c0580 -7922> 2014-11-08 18:14:38.347620 7fc5a152e700 5 -- op tracker -- seq: 46869, time: 2014-11-08 18:14:38.347549, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -7921> 2014-11-08 18:14:38.347627 7fc5a152e700 5 -- op tracker -- seq: 46869, time: 2014-11-08 18:14:38.347550, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -7920> 2014-11-08 18:14:38.347634 7fc5a152e700 5 -- op tracker -- seq: 46869, time: 2014-11-08 18:14:38.347602, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -7919> 2014-11-08 18:14:38.347640 7fc5a152e700 5 -- op tracker -- seq: 46869, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -7918> 2014-11-08 18:14:38.347663 7fc5aeb13700 5 -- op tracker -- seq: 46869, time: 2014-11-08 18:14:38.347663, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -7917> 2014-11-08 18:14:38.347706 7fc5aeb13700 5 -- op tracker -- seq: 46869, time: 2014-11-08 18:14:38.347706, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -7916> 2014-11-08 18:14:38.423555 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59242, attrs_read=0)) v1 -- ?+0 0x1c89b480 con 0x21468c60 -7915> 2014-11-08 18:14:38.423613 7fc5af314700 5 -- op tracker -- seq: 46868, time: 2014-11-08 18:14:38.423613, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59242, to_read={8210414e/1000002a6d4.00000015/head//4=0,4194304}, attrs_to_read=)) -7914> 2014-11-08 18:14:38.671823 7fc5c362f700 5 osd.9 5773 tick -7913> 2014-11-08 18:14:38.941512 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -7912> 2014-11-08 18:14:38.941546 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1d3210e0 con 0x1cae2940 -7911> 2014-11-08 18:14:38.941569 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x25fc7800 con 0x1cae2680 -7910> 2014-11-08 18:14:38.941579 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1d1b5280 con 0x1c5c02c0 -7909> 2014-11-08 18:14:38.941588 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x211e7800 con 0x1cae0160 -7908> 2014-11-08 18:14:38.941596 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x211e0b40 con 0x1cae0f20 -7907> 2014-11-08 18:14:38.941605 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x211e7bc0 con 0x1cae0c60 -7906> 2014-11-08 18:14:38.941614 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1d327800 con 0x1cae23c0 -7905> 2014-11-08 18:14:38.941622 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1d327bc0 con 0x1cae56a0 -7904> 2014-11-08 18:14:38.941631 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1caaa580 con 0x1cbf9ce0 -7903> 2014-11-08 18:14:38.941640 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x2290a760 con 0x1d603020 -7902> 2014-11-08 18:14:38.941648 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x22908f00 con 0x1dbd9b80 -7901> 2014-11-08 18:14:38.941657 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1ddcd0a0 con 0x1d235d80 -7900> 2014-11-08 18:14:38.941665 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x211e4560 con 0x1cae1fa0 -7899> 2014-11-08 18:14:38.941674 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x1c834560 con 0x1cae1ce0 -7898> 2014-11-08 18:14:38.941682 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x2371d640 con 0x1cae1340 -7897> 2014-11-08 18:14:38.941698 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x22585dc0 con 0x1cae1080 -7896> 2014-11-08 18:14:38.941707 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x22e9d640 con 0x1cae1a20 -7895> 2014-11-08 18:14:38.941715 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:38.941542) v2 -- ?+0 0x211e41a0 con 0x1cae2c00 -7894> 2014-11-08 18:14:38.942794 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x259f1a40 con 0x1cae0f20 -7893> 2014-11-08 18:14:38.942819 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1e897080 con 0x1cae0c60 -7892> 2014-11-08 18:14:38.942828 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x25f96720 con 0x1cae1fa0 -7891> 2014-11-08 18:14:38.942838 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x4534ec0 con 0x1cae1ce0 -7890> 2014-11-08 18:14:38.942846 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1d4f03c0 con 0x1cae2940 -7889> 2014-11-08 18:14:38.942855 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1f4c14a0 con 0x1cae2680 -7888> 2014-11-08 18:14:38.942863 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x2515d0a0 con 0x1cae23c0 -7887> 2014-11-08 18:14:38.942872 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x2583c740 con 0x1cae56a0 -7886> 2014-11-08 18:14:38.942879 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 497 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x246cb2a0 con 0x1cbf9ce0 -7885> 2014-11-08 18:14:38.942889 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 497 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1dbf83c0 con 0x1d603020 -7884> 2014-11-08 18:14:38.942897 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 304 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1db19fe0 con 0x1dbd9b80 -7883> 2014-11-08 18:14:38.942905 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x257c41a0 con 0x1c5c02c0 -7882> 2014-11-08 18:14:38.942914 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x2290cce0 con 0x1cae0160 -7881> 2014-11-08 18:14:38.946247 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 304 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x23d74380 con 0x1d235d80 -7880> 2014-11-08 18:14:38.946268 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x25e83a20 con 0x1cae1340 -7879> 2014-11-08 18:14:38.946278 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1deccec0 con 0x1cae1080 -7878> 2014-11-08 18:14:38.946299 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1d326900 con 0x1cae1a20 -7877> 2014-11-08 18:14:38.946309 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3717 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:38.941542) v2 ==== 47+0+0 (3336396803 0 0) 0x1dbfddc0 con 0x1cae2c00 -7876> 2014-11-08 18:14:39.599624 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3670 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:39.597259) v2 ==== 47+0+0 (1950340917 0 0) 0x224a1fe0 con 0x1cce2ec0 -7875> 2014-11-08 18:14:39.599675 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:39.597259) v2 -- ?+0 0x257c7bc0 con 0x1cce2ec0 -7874> 2014-11-08 18:14:39.599632 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3670 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:39.597259) v2 ==== 47+0+0 (1950340917 0 0) 0x23146180 con 0x1cce3440 -7873> 2014-11-08 18:14:39.599715 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:39.597259) v2 -- ?+0 0x2026a580 con 0x1cce3440 -7872> 2014-11-08 18:14:39.671946 7fc5c362f700 5 osd.9 5773 tick -7871> 2014-11-08 18:14:39.921922 7fc5b2b1b700 10 monclient: tick -7870> 2014-11-08 18:14:39.921940 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:14:09.921938) -7869> 2014-11-08 18:14:39.921947 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:14:39.921947; renew after: 2014-11-08 18:16:29.290049) -- no -7868> 2014-11-08 18:14:40.092408 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2202 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) v2 ==== 370+0+0 (2196953512 0 0) 0x1d5b2200 con 0x1e50aec0 -7867> 2014-11-08 18:14:40.092441 7fc5a56ef700 5 -- op tracker -- seq: 46870, time: 2014-11-08 18:14:40.092306, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) -7866> 2014-11-08 18:14:40.092456 7fc5a56ef700 5 -- op tracker -- seq: 46870, time: 2014-11-08 18:14:40.092310, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) -7865> 2014-11-08 18:14:40.092463 7fc5a56ef700 5 -- op tracker -- seq: 46870, time: 2014-11-08 18:14:40.092400, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) -7864> 2014-11-08 18:14:40.092470 7fc5a56ef700 5 -- op tracker -- seq: 46870, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) -7863> 2014-11-08 18:14:40.092510 7fc5ac30e700 5 -- op tracker -- seq: 46870, time: 2014-11-08 18:14:40.092509, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) -7862> 2014-11-08 18:14:40.092538 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2203 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) v2 ==== 240+0+0 (3479755939 0 0) 0x1d85c200 con 0x1e50aec0 -7861> 2014-11-08 18:14:40.092548 7fc5a56ef700 5 -- op tracker -- seq: 46871, time: 2014-11-08 18:14:40.092501, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) -7860> 2014-11-08 18:14:40.092553 7fc5a56ef700 5 -- op tracker -- seq: 46871, time: 2014-11-08 18:14:40.092502, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) -7859> 2014-11-08 18:14:40.092564 7fc5a56ef700 5 -- op tracker -- seq: 46871, time: 2014-11-08 18:14:40.092528, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) -7858> 2014-11-08 18:14:40.092568 7fc5a56ef700 5 -- op tracker -- seq: 46871, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) -7857> 2014-11-08 18:14:40.092589 7fc5ad310700 5 -- op tracker -- seq: 46871, time: 2014-11-08 18:14:40.092589, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) -7856> 2014-11-08 18:14:40.092647 7fc5ac30e700 5 -- op tracker -- seq: 46870, time: 2014-11-08 18:14:40.092647, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(ca5df418/1000001b502.00000378/head//4),PushReplyOp(246df418/1000002bdbc.000000fe/head//4),PushReplyOp(611ef418/1000002a661.00000011/head//4),PushReplyOp(8a2ef418/10000019d91.00000026/head//4),PushReplyOp(318ef418/1000002a454.00000037/head//4)]) -7855> 2014-11-08 18:14:40.092874 7fc5a56ef700 5 -- op tracker -- seq: 46871, time: 2014-11-08 18:14:40.092874, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(78ebd484/1000001a34e.00000036/head//4),PushReplyOp(586cd484/1000002bca6.0000008e/head//4),PushReplyOp(516dd484/10000029456.00000006/head//4)]) -7854> 2014-11-08 18:14:40.097887 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12045 ==== MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (848150622 0 0) 0x1ce61f80 con 0x1c5c2680 -7853> 2014-11-08 18:14:40.097905 7fc5a6cfe700 5 -- op tracker -- seq: 46872, time: 2014-11-08 18:14:40.097817, event: header_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) -7852> 2014-11-08 18:14:40.097914 7fc5a6cfe700 5 -- op tracker -- seq: 46872, time: 2014-11-08 18:14:40.097819, event: throttled, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) -7851> 2014-11-08 18:14:40.097922 7fc5a6cfe700 5 -- op tracker -- seq: 46872, time: 2014-11-08 18:14:40.097882, event: all_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) -7850> 2014-11-08 18:14:40.097929 7fc5a6cfe700 5 -- op tracker -- seq: 46872, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) -7849> 2014-11-08 18:14:40.097956 7fc5afb15700 5 -- op tracker -- seq: 46872, time: 2014-11-08 18:14:40.097956, event: reached_pg, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) -7848> 2014-11-08 18:14:40.174642 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=fa3ed484/1000002a7de.00000070/head//4,1f4fd484/1000002b091.00000098/head//4,2b7fd484/1000001a64d.00000095/head//4,ddafd484/1000001950e.00000013/head//4,f9bfd484/1000002a825.000000bd/head//4)) v1 -- ?+0 0x224ada00 con 0x1c5c0580 -7847> 2014-11-08 18:14:40.174677 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x224ac800 con 0x42c86e0 -7846> 2014-11-08 18:14:40.174700 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 6dfd484/1000001abf1.00000009/head//4 -7845> 2014-11-08 18:14:40.174726 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb 516dd484/10000029456.00000006/head//4) v3 -- ?+0 0x1d868000 con 0x1e50aec0 -7844> 2014-11-08 18:14:40.175039 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x224ac800 con 0x42c86e0 -7843> 2014-11-08 18:14:40.175058 7fc5b9328700 5 -- op tracker -- seq: 46873, time: 2014-11-08 18:14:40.174693, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) -7842> 2014-11-08 18:14:40.175069 7fc5b9328700 5 -- op tracker -- seq: 46873, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) -7841> 2014-11-08 18:14:40.175083 7fc5b9328700 5 -- op tracker -- seq: 46873, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) -7840> 2014-11-08 18:14:40.175089 7fc5b9328700 5 -- op tracker -- seq: 46873, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) -7839> 2014-11-08 18:14:40.175125 7fc5ad310700 5 -- op tracker -- seq: 46873, time: 2014-11-08 18:14:40.175125, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) -7838> 2014-11-08 18:14:40.401470 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=2ddff418/1000002845d.00000061/head//4,26810518/1000001a239.00000407/head//4,20020518/1000001aac2.0000001a/head//4)) v1 -- ?+0 0x224af740 con 0x1c5c0580 -7837> 2014-11-08 18:14:40.401516 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x224adc40 con 0x42c86e0 -7836> 2014-11-08 18:14:40.401532 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 8c820518/10000019a25.000003f3/head//4 -7835> 2014-11-08 18:14:40.401562 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb 318ef418/1000002a454.00000037/head//4) v3 -- ?+0 0x1d3ca880 con 0x1e50aec0 -7834> 2014-11-08 18:14:40.401719 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x224adc40 con 0x42c86e0 -7833> 2014-11-08 18:14:40.401735 7fc5b9328700 5 -- op tracker -- seq: 46874, time: 2014-11-08 18:14:40.401526, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) -7832> 2014-11-08 18:14:40.401746 7fc5b9328700 5 -- op tracker -- seq: 46874, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) -7831> 2014-11-08 18:14:40.401752 7fc5b9328700 5 -- op tracker -- seq: 46874, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) -7830> 2014-11-08 18:14:40.401757 7fc5b9328700 5 -- op tracker -- seq: 46874, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) -7829> 2014-11-08 18:14:40.401871 7fc5ac30e700 5 -- op tracker -- seq: 46874, time: 2014-11-08 18:14:40.401871, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) -7828> 2014-11-08 18:14:40.672056 7fc5c362f700 5 osd.9 5773 tick -7827> 2014-11-08 18:14:40.672102 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -7826> 2014-11-08 18:14:40.672114 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(3 pgs tid 1326 v 0) v1 -- ?+0 0x22d7f640 con 0x44ed6a0 -7825> 2014-11-08 18:14:40.715434 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) v1 -- ?+0 0x44a5a00 con 0x42c86e0 -7824> 2014-11-08 18:14:40.715493 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x44a5a00 con 0x42c86e0 -7823> 2014-11-08 18:14:40.715493 7fc5ad310700 5 -- op tracker -- seq: 46873, time: 2014-11-08 18:14:40.715492, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4404, to_read={fa3ed484/1000002a7de.00000070/head//4=0,4194304,1f4fd484/1000002b091.00000098/head//4=0,4194304,2b7fd484/1000001a64d.00000095/head//4=0,4194304,ddafd484/1000001950e.00000013/head//4=0,4194304,f9bfd484/1000002a825.000000bd/head//4=0,4194304}, attrs_to_read=)) -7822> 2014-11-08 18:14:40.715511 7fc5b9328700 5 -- op tracker -- seq: 46875, time: 2014-11-08 18:14:40.715482, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) -7821> 2014-11-08 18:14:40.715524 7fc5b9328700 5 -- op tracker -- seq: 46875, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) -7820> 2014-11-08 18:14:40.715528 7fc5b9328700 5 -- op tracker -- seq: 46875, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) -7819> 2014-11-08 18:14:40.715532 7fc5b9328700 5 -- op tracker -- seq: 46875, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) -7818> 2014-11-08 18:14:40.715580 7fc5ad310700 5 -- op tracker -- seq: 46875, time: 2014-11-08 18:14:40.715580, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) -7817> 2014-11-08 18:14:40.715724 7fc5ad310700 5 -- op tracker -- seq: 46875, time: 2014-11-08 18:14:40.715724, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=0)) -7816> 2014-11-08 18:14:40.746059 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) v1 -- ?+0 0x1c89bc00 con 0x42c86e0 -7815> 2014-11-08 18:14:40.746105 7fc5ac30e700 5 -- op tracker -- seq: 46874, time: 2014-11-08 18:14:40.746105, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4405, to_read={2ddff418/1000002845d.00000061/head//4=0,4194304,26810518/1000001a239.00000407/head//4=0,4194304,20020518/1000001aac2.0000001a/head//4=0,4194304}, attrs_to_read=)) -7814> 2014-11-08 18:14:40.746129 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1c89bc00 con 0x42c86e0 -7813> 2014-11-08 18:14:40.746145 7fc5b9328700 5 -- op tracker -- seq: 46876, time: 2014-11-08 18:14:40.746095, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) -7812> 2014-11-08 18:14:40.746149 7fc5b9328700 5 -- op tracker -- seq: 46876, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) -7811> 2014-11-08 18:14:40.746151 7fc5b9328700 5 -- op tracker -- seq: 46876, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) -7810> 2014-11-08 18:14:40.746153 7fc5b9328700 5 -- op tracker -- seq: 46876, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) -7809> 2014-11-08 18:14:40.746186 7fc5aeb13700 5 -- op tracker -- seq: 46876, time: 2014-11-08 18:14:40.746186, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) -7808> 2014-11-08 18:14:40.746236 7fc5aeb13700 5 -- op tracker -- seq: 46876, time: 2014-11-08 18:14:40.746236, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=0)) -7807> 2014-11-08 18:14:40.843992 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3704 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:40.841511) v2 ==== 47+0+0 (3956956232 0 0) 0x2495bfc0 con 0x1c8b7d20 -7806> 2014-11-08 18:14:40.844037 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:40.841511) v2 -- ?+0 0x23146180 con 0x1c8b7d20 -7805> 2014-11-08 18:14:40.844070 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3704 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:40.841511) v2 ==== 47+0+0 (3956956232 0 0) 0x1f4c4740 con 0x1c9f8580 -7804> 2014-11-08 18:14:40.844081 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:40.841511) v2 -- ?+0 0x224a1fe0 con 0x1c9f8580 -7803> 2014-11-08 18:14:40.928583 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.bbs0 5773 ECSubReadReply(tid=25183, attrs_read=0)) v1 -- ?+0 0x211d6e00 con 0x1c5c2680 -7802> 2014-11-08 18:14:40.936163 7fc5afb15700 5 -- op tracker -- seq: 46872, time: 2014-11-08 18:14:40.936163, event: done, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25183, to_read={b5ad90bb/1000001a63a.0000000a/head//4=0,4194304,d88e90bb/100000176d7.00000000/head//4=0,4194304,f9ff90bb/1000002a7fd.00000077/head//4=0,4194304,4130a0bb/1000002bcd7.000000ac/head//4=0,4194304,9ff1a0bb/10000019717.00000019/head//4=0,4194304}, attrs_to_read=)) -7801> 2014-11-08 18:14:40.960439 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8490 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) v1 ==== 10488220+0+0 (3787492606 0 0) 0x21838780 con 0x1c5c0580 -7800> 2014-11-08 18:14:40.960481 7fc5a152e700 5 -- op tracker -- seq: 46877, time: 2014-11-08 18:14:40.720952, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) -7799> 2014-11-08 18:14:40.960491 7fc5a152e700 5 -- op tracker -- seq: 46877, time: 2014-11-08 18:14:40.720954, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) -7798> 2014-11-08 18:14:40.960494 7fc5a152e700 5 -- op tracker -- seq: 46877, time: 2014-11-08 18:14:40.960393, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) -7797> 2014-11-08 18:14:40.960499 7fc5a152e700 5 -- op tracker -- seq: 46877, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) -7796> 2014-11-08 18:14:40.960543 7fc5ad310700 5 -- op tracker -- seq: 46877, time: 2014-11-08 18:14:40.960543, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) -7795> 2014-11-08 18:14:40.986165 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(fa3ed484/1000002a7de.00000070/head//4, version: 3458'5396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fa3ed484/1000002a7de.00000070/head//4@3458'5396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1f4fd484/1000002b091.00000098/head//4, version: 4157'7777, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1f4fd484/1000002b091.00000098/head//4@4157'7777, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b7fd484/1000001a64d.00000095/head//4, version: 397'1951, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b7fd484/1000001a64d.00000095/head//4@397'1951, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ddafd484/1000001950e.00000013/head//4, version: 397'578, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ddafd484/ 1000001950e.00000013/head//4@397'578, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f9bfd484/1000002a825.000000bd/head//4, version: 3458'5431, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f9bfd484/1000002a825.000000bd/head//4@3458'5431, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9200 con 0x1e50aec0 -7794> 2014-11-08 18:14:40.986299 7fc5ad310700 5 -- op tracker -- seq: 46877, time: 2014-11-08 18:14:40.986298, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4404, attrs_read=5)) -7793> 2014-11-08 18:14:41.087506 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1501 ==== pg_stats_ack(3 pgs tid 1326) v1 ==== 91+0+0 (3767562893 0 0) 0x2286ad00 con 0x44ed6a0 -7792> 2014-11-08 18:14:41.129531 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8491 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) v1 ==== 6292958+0+0 (4243145999 0 0) 0x21838a00 con 0x1c5c0580 -7791> 2014-11-08 18:14:41.129571 7fc5a152e700 5 -- op tracker -- seq: 46878, time: 2014-11-08 18:14:40.960646, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) -7790> 2014-11-08 18:14:41.129585 7fc5a152e700 5 -- op tracker -- seq: 46878, time: 2014-11-08 18:14:40.960648, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) -7789> 2014-11-08 18:14:41.129595 7fc5a152e700 5 -- op tracker -- seq: 46878, time: 2014-11-08 18:14:41.129500, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) -7788> 2014-11-08 18:14:41.129603 7fc5a152e700 5 -- op tracker -- seq: 46878, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) -7787> 2014-11-08 18:14:41.129649 7fc5ac30e700 5 -- op tracker -- seq: 46878, time: 2014-11-08 18:14:41.129649, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) -7786> 2014-11-08 18:14:41.145355 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(2ddff418/1000002845d.00000061/head//4, version: 3033'3841, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2ddff418/1000002845d.00000061/head//4@3033'3841, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(26810518/1000001a239.00000407/head//4, version: 397'1991, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26810518/1000001a239.00000407/head//4@397'1991, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(20020518/1000001aac2.0000001a/head//4, version: 2829'2558, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(20020518/1000001aac2.0000001a/head//4@2829'2558, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d5b2200 con 0x1e50aec0 -7785> 2014-11-08 18:14:41.145491 7fc5ac30e700 5 -- op tracker -- seq: 46878, time: 2014-11-08 18:14:41.145491, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4405, attrs_read=3)) -7784> 2014-11-08 18:14:41.396314 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 492 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:41.392707) v2 ==== 47+0+0 (1938063765 0 0) 0x24885460 con 0x1eb9c8e0 -7783> 2014-11-08 18:14:41.396344 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:41.392707) v2 -- ?+0 0x2495bfc0 con 0x1eb9c8e0 -7782> 2014-11-08 18:14:41.396419 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 492 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:41.392707) v2 ==== 47+0+0 (1938063765 0 0) 0x22e5b660 con 0x1eb9d800 -7781> 2014-11-08 18:14:41.396429 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:41.392707) v2 -- ?+0 0x1f4c4740 con 0x1eb9d800 -7780> 2014-11-08 18:14:41.506281 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2204 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x213bfbc0 con 0x1e50aec0 -7779> 2014-11-08 18:14:41.506321 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506176, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -7778> 2014-11-08 18:14:41.506330 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506180, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -7777> 2014-11-08 18:14:41.506332 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506259, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -7776> 2014-11-08 18:14:41.506335 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506316, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -7775> 2014-11-08 18:14:41.506338 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506338, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -7774> 2014-11-08 18:14:41.506347 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506347, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -7773> 2014-11-08 18:14:41.506353 7fc5b9b29700 5 -- op tracker -- seq: 46879, time: 2014-11-08 18:14:41.506353, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -7772> 2014-11-08 18:14:41.506409 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2205 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x22858b40 con 0x1e50aec0 -7771> 2014-11-08 18:14:41.506419 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506379, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -7770> 2014-11-08 18:14:41.506422 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506379, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -7769> 2014-11-08 18:14:41.506425 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506401, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -7768> 2014-11-08 18:14:41.506427 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506417, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -7767> 2014-11-08 18:14:41.506430 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506430, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -7766> 2014-11-08 18:14:41.506434 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506434, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -7765> 2014-11-08 18:14:41.506437 7fc5b9b29700 5 -- op tracker -- seq: 46880, time: 2014-11-08 18:14:41.506437, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -7764> 2014-11-08 18:14:41.506508 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2206 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x227e05a0 con 0x1e50aec0 -7763> 2014-11-08 18:14:41.506535 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506479, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -7762> 2014-11-08 18:14:41.506538 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506479, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -7761> 2014-11-08 18:14:41.506540 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506500, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -7760> 2014-11-08 18:14:41.506543 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506533, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -7759> 2014-11-08 18:14:41.506546 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506546, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -7758> 2014-11-08 18:14:41.506549 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506549, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -7757> 2014-11-08 18:14:41.506553 7fc5b9b29700 5 -- op tracker -- seq: 46881, time: 2014-11-08 18:14:41.506553, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -7756> 2014-11-08 18:14:41.506597 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2207 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x2495e360 con 0x1e50aec0 -7755> 2014-11-08 18:14:41.506607 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506568, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -7754> 2014-11-08 18:14:41.506610 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506569, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -7753> 2014-11-08 18:14:41.506613 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506589, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -7752> 2014-11-08 18:14:41.506615 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506605, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -7751> 2014-11-08 18:14:41.506618 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506618, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -7750> 2014-11-08 18:14:41.506622 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506622, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -7749> 2014-11-08 18:14:41.506625 7fc5b9b29700 5 -- op tracker -- seq: 46882, time: 2014-11-08 18:14:41.506625, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -7748> 2014-11-08 18:14:41.506670 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2208 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x2495f080 con 0x1e50aec0 -7747> 2014-11-08 18:14:41.506680 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506641, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -7746> 2014-11-08 18:14:41.506683 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506642, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -7745> 2014-11-08 18:14:41.506694 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506662, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -7744> 2014-11-08 18:14:41.506696 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506679, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -7743> 2014-11-08 18:14:41.506727 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506727, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -7742> 2014-11-08 18:14:41.506732 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506732, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -7741> 2014-11-08 18:14:41.506735 7fc5b9b29700 5 -- op tracker -- seq: 46883, time: 2014-11-08 18:14:41.506735, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -7740> 2014-11-08 18:14:41.528263 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(a06b90b7/10000019b73.0000002a/head//4),PushReplyOp(dfcb90b7/1000001b501.00000ad0/head//4)]) v2 -- ?+0 0x21613c00 con 0x1e50aec0 -7739> 2014-11-08 18:14:41.535314 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(1687fcb5/1000002b03d.00000046/head//4)]) v2 -- ?+0 0x1d5b1800 con 0x1c5c0580 -7738> 2014-11-08 18:14:41.538025 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8492 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) v3 ==== 757+0+0 (3223089608 0 0) 0x1d33e780 con 0x1c5c0580 -7737> 2014-11-08 18:14:41.538062 7fc5a152e700 5 -- op tracker -- seq: 46884, time: 2014-11-08 18:14:41.537930, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7736> 2014-11-08 18:14:41.538070 7fc5a152e700 5 -- op tracker -- seq: 46884, time: 2014-11-08 18:14:41.537935, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7735> 2014-11-08 18:14:41.538073 7fc5a152e700 5 -- op tracker -- seq: 46884, time: 2014-11-08 18:14:41.538016, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7734> 2014-11-08 18:14:41.538077 7fc5a152e700 5 -- op tracker -- seq: 46884, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7733> 2014-11-08 18:14:41.538157 7fc5adb11700 5 -- op tracker -- seq: 46884, time: 2014-11-08 18:14:41.538157, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7732> 2014-11-08 18:14:41.538169 7fc5adb11700 5 -- op tracker -- seq: 46884, time: 2014-11-08 18:14:41.538169, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7731> 2014-11-08 18:14:41.538219 7fc5adb11700 5 -- op tracker -- seq: 46884, time: 2014-11-08 18:14:41.538219, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 1687fcb5/1000002b03d.00000046/head//4) -7730> 2014-11-08 18:14:41.542550 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(253d5b16/10000019b04.000002ef/head//4),PushReplyOp(134d5b16/1000002b8f2.00000010/head//4),PushReplyOp(6cbd5b16/1000002b0e6.00000011/head//4),PushReplyOp(f9dd5b16/1000002af7c.000000b7/head//4),PushReplyOp(d0ed5b16/1000001b3fe.0000001f/head//4)]) v2 -- ?+0 0x1d5b4e00 con 0x1c5c0580 -7729> 2014-11-08 18:14:41.546318 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8493 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) v3 ==== 757+0+0 (914579369 0 0) 0x1d651200 con 0x1c5c0580 -7728> 2014-11-08 18:14:41.546376 7fc5a152e700 5 -- op tracker -- seq: 46885, time: 2014-11-08 18:14:41.546155, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7727> 2014-11-08 18:14:41.546386 7fc5a152e700 5 -- op tracker -- seq: 46885, time: 2014-11-08 18:14:41.546163, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7726> 2014-11-08 18:14:41.546389 7fc5a152e700 5 -- op tracker -- seq: 46885, time: 2014-11-08 18:14:41.546304, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7725> 2014-11-08 18:14:41.546459 7fc5a152e700 5 -- op tracker -- seq: 46885, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7724> 2014-11-08 18:14:41.546515 7fc5afb15700 5 -- op tracker -- seq: 46885, time: 2014-11-08 18:14:41.546515, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7723> 2014-11-08 18:14:41.546528 7fc5afb15700 5 -- op tracker -- seq: 46885, time: 2014-11-08 18:14:41.546528, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7722> 2014-11-08 18:14:41.546574 7fc5afb15700 5 -- op tracker -- seq: 46885, time: 2014-11-08 18:14:41.546574, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d0ed5b16/1000001b3fe.0000001f/head//4) -7721> 2014-11-08 18:14:41.546996 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.5ds1 5773 [PushReplyOp(4bcbbddd/1000002a94c.0000009e/head//4)]) v2 -- ?+0 0x1ebb5c00 con 0x21468c60 -7720> 2014-11-08 18:14:41.672200 7fc5c362f700 5 osd.9 5773 tick -7719> 2014-11-08 18:14:41.690077 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2209 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x2495e540 con 0x1e50aec0 -7718> 2014-11-08 18:14:41.690114 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.689981, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -7717> 2014-11-08 18:14:41.690122 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.689984, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -7716> 2014-11-08 18:14:41.690125 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.690050, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -7715> 2014-11-08 18:14:41.690127 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.690109, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -7714> 2014-11-08 18:14:41.690130 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.690130, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -7713> 2014-11-08 18:14:41.690138 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.690138, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -7712> 2014-11-08 18:14:41.690144 7fc5b9b29700 5 -- op tracker -- seq: 46886, time: 2014-11-08 18:14:41.690144, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -7711> 2014-11-08 18:14:41.690195 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2210 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x21225280 con 0x1e50aec0 -7710> 2014-11-08 18:14:41.690204 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690166, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -7709> 2014-11-08 18:14:41.690208 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690166, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -7708> 2014-11-08 18:14:41.690210 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690187, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -7707> 2014-11-08 18:14:41.690213 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690203, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -7706> 2014-11-08 18:14:41.690215 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690215, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -7705> 2014-11-08 18:14:41.690219 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690219, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -7704> 2014-11-08 18:14:41.690223 7fc5b9b29700 5 -- op tracker -- seq: 46887, time: 2014-11-08 18:14:41.690223, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -7703> 2014-11-08 18:14:41.690303 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2211 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1d4f1fe0 con 0x1e50aec0 -7702> 2014-11-08 18:14:41.690313 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690238, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -7701> 2014-11-08 18:14:41.690316 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690239, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -7700> 2014-11-08 18:14:41.690319 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690294, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -7699> 2014-11-08 18:14:41.690321 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690312, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -7698> 2014-11-08 18:14:41.690324 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690324, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -7697> 2014-11-08 18:14:41.690328 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690328, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -7696> 2014-11-08 18:14:41.690331 7fc5b9b29700 5 -- op tracker -- seq: 46888, time: 2014-11-08 18:14:41.690331, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -7695> 2014-11-08 18:14:41.723222 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8494 ==== MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3030761998 0 0) 0x1d5b4e00 con 0x1c5c0580 -7694> 2014-11-08 18:14:41.723271 7fc5a152e700 5 -- op tracker -- seq: 46889, time: 2014-11-08 18:14:41.699715, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7693> 2014-11-08 18:14:41.723290 7fc5a152e700 5 -- op tracker -- seq: 46889, time: 2014-11-08 18:14:41.699719, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7692> 2014-11-08 18:14:41.723303 7fc5a152e700 5 -- op tracker -- seq: 46889, time: 2014-11-08 18:14:41.723194, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7691> 2014-11-08 18:14:41.723315 7fc5a152e700 5 -- op tracker -- seq: 46889, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7690> 2014-11-08 18:14:41.723361 7fc5ad310700 5 -- op tracker -- seq: 46889, time: 2014-11-08 18:14:41.723361, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7689> 2014-11-08 18:14:41.729734 7fc5a152e700 5 -- op tracker -- seq: 46889, time: 2014-11-08 18:14:41.729734, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(4d3e5b16/1000001aaa8.00000026/head//4, version: 2829'2414, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4d3e5b16/1000001aaa8.00000026/head//4@2829'2414, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7688> 2014-11-08 18:14:41.792541 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 314 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:41.789919) v2 ==== 47+0+0 (3669912345 0 0) 0x2026f260 con 0x1eb9c780 -7687> 2014-11-08 18:14:41.792572 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 314 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:41.789919) v2 ==== 47+0+0 (3669912345 0 0) 0x1d8bbfc0 con 0x1eb9d6a0 -7686> 2014-11-08 18:14:41.792590 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:41.789919) v2 -- ?+0 0x24885460 con 0x1eb9c780 -7685> 2014-11-08 18:14:41.792753 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:41.789919) v2 -- ?+0 0x22e5b660 con 0x1eb9d6a0 -7684> 2014-11-08 18:14:41.814279 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3734 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:41.813772) v2 ==== 47+0+0 (3268311887 0 0) 0x23682ee0 con 0x1cce3180 -7683> 2014-11-08 18:14:41.814303 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:41.813772) v2 -- ?+0 0x1d8bbfc0 con 0x1cce3180 -7682> 2014-11-08 18:14:41.814348 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3734 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:41.813772) v2 ==== 47+0+0 (3268311887 0 0) 0x2286a940 con 0x1cce3700 -7681> 2014-11-08 18:14:41.814358 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:41.813772) v2 -- ?+0 0x2026f260 con 0x1cce3700 -7680> 2014-11-08 18:14:41.822878 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1eae5dc0 con 0x1c5c0580 -7679> 2014-11-08 18:14:42.160674 7fc5a5ff8700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.16:6804/1238 pipe(0x215a3c80 sd=24 :6800 s=2 pgs=1750 cs=1 l=1 c=0x259953e0).reader couldn't read tag, (0) Success -7678> 2014-11-08 18:14:42.160769 7fc5a5ff8700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.16:6804/1238 pipe(0x215a3c80 sd=24 :6800 s=2 pgs=1750 cs=1 l=1 c=0x259953e0).fault (0) Success -7677> 2014-11-08 18:14:42.160902 7fc5bb32c700 1 osd.9 5773 ms_handle_reset con 0x259953e0 session 0x1d5f9b80 -7676> 2014-11-08 18:14:42.162091 7fc5a5ff8700 1 -- 192.168.1.18:6800/4928 >> :/0 pipe(0x2244e5c0 sd=24 :6800 s=0 pgs=0 cs=0 l=0 c=0x25996720).accept sd=24 192.168.1.16:39258/0 -7675> 2014-11-08 18:14:42.304797 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8495 ==== MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3619703487 0 0) 0x1d5b4e00 con 0x1c5c0580 -7674> 2014-11-08 18:14:42.304848 7fc5a152e700 5 -- op tracker -- seq: 46890, time: 2014-11-08 18:14:42.271777, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7673> 2014-11-08 18:14:42.304866 7fc5a152e700 5 -- op tracker -- seq: 46890, time: 2014-11-08 18:14:42.271782, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7672> 2014-11-08 18:14:42.304879 7fc5a152e700 5 -- op tracker -- seq: 46890, time: 2014-11-08 18:14:42.304771, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7671> 2014-11-08 18:14:42.304892 7fc5a152e700 5 -- op tracker -- seq: 46890, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7670> 2014-11-08 18:14:42.304956 7fc5b0316700 5 -- op tracker -- seq: 46890, time: 2014-11-08 18:14:42.304956, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7669> 2014-11-08 18:14:42.305707 7fc5a152e700 5 -- op tracker -- seq: 46890, time: 2014-11-08 18:14:42.305707, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(66f7fcb5/1000002a5b1.00000008/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66f7fcb5/1000002a5b1.00000008/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7668> 2014-11-08 18:14:42.389175 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1eae6ea0 con 0x1c5c0580 -7667> 2014-11-08 18:14:42.542908 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.8fs0 5773 [PushReplyOp(12e8678f/1000002af7f.0000001b/head//4),PushReplyOp(5439678f/1000001a8e7.0000001c/head//4),PushReplyOp(eb79678f/1000002bbec.00000004/head//4),PushReplyOp(76aa678f/1000002a47e.00000015/head//4)]) v2 -- ?+0 0x24847200 con 0x21468c60 -7666> 2014-11-08 18:14:42.544638 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3705 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:42.542435) v2 ==== 47+0+0 (3759610352 0 0) 0x1f4c5dc0 con 0x1c9f8580 -7665> 2014-11-08 18:14:42.544696 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:42.542435) v2 -- ?+0 0x23682ee0 con 0x1c9f8580 -7664> 2014-11-08 18:14:42.545443 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3705 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:42.542435) v2 ==== 47+0+0 (3759610352 0 0) 0x2495f800 con 0x1c8b7d20 -7663> 2014-11-08 18:14:42.545457 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:42.542435) v2 -- ?+0 0x2286a940 con 0x1c8b7d20 -7662> 2014-11-08 18:14:42.545483 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(4d3e5b16/1000001aaa8.00000026/head//4)]) v2 -- ?+0 0x229bc200 con 0x1c5c0580 -7661> 2014-11-08 18:14:42.547060 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(66f7fcb5/1000002a5b1.00000008/head//4)]) v2 -- ?+0 0x1d3af400 con 0x1c5c0580 -7660> 2014-11-08 18:14:42.549859 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8496 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) v3 ==== 757+0+0 (3163114633 0 0) 0x1d33b600 con 0x1c5c0580 -7659> 2014-11-08 18:14:42.549891 7fc5a152e700 5 -- op tracker -- seq: 46891, time: 2014-11-08 18:14:42.549794, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7658> 2014-11-08 18:14:42.549901 7fc5a152e700 5 -- op tracker -- seq: 46891, time: 2014-11-08 18:14:42.549796, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7657> 2014-11-08 18:14:42.549904 7fc5a152e700 5 -- op tracker -- seq: 46891, time: 2014-11-08 18:14:42.549854, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7656> 2014-11-08 18:14:42.549909 7fc5a152e700 5 -- op tracker -- seq: 46891, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7655> 2014-11-08 18:14:42.549941 7fc5afb15700 5 -- op tracker -- seq: 46891, time: 2014-11-08 18:14:42.549941, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7654> 2014-11-08 18:14:42.549952 7fc5afb15700 5 -- op tracker -- seq: 46891, time: 2014-11-08 18:14:42.549951, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7653> 2014-11-08 18:14:42.549963 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8497 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) v3 ==== 757+0+0 (1393942396 0 0) 0x1d33a400 con 0x1c5c0580 -7652> 2014-11-08 18:14:42.549971 7fc5a152e700 5 -- op tracker -- seq: 46892, time: 2014-11-08 18:14:42.549933, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7651> 2014-11-08 18:14:42.549975 7fc5a152e700 5 -- op tracker -- seq: 46892, time: 2014-11-08 18:14:42.549933, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7650> 2014-11-08 18:14:42.549978 7fc5a152e700 5 -- op tracker -- seq: 46892, time: 2014-11-08 18:14:42.549960, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7649> 2014-11-08 18:14:42.549981 7fc5a152e700 5 -- op tracker -- seq: 46892, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7648> 2014-11-08 18:14:42.549995 7fc5afb15700 5 -- op tracker -- seq: 46891, time: 2014-11-08 18:14:42.549994, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4d3e5b16/1000001aaa8.00000026/head//4) -7647> 2014-11-08 18:14:42.550015 7fc5adb11700 5 -- op tracker -- seq: 46892, time: 2014-11-08 18:14:42.550015, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7646> 2014-11-08 18:14:42.550020 7fc5adb11700 5 -- op tracker -- seq: 46892, time: 2014-11-08 18:14:42.550020, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7645> 2014-11-08 18:14:42.550046 7fc5adb11700 5 -- op tracker -- seq: 46892, time: 2014-11-08 18:14:42.550046, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 66f7fcb5/1000002a5b1.00000008/head//4) -7644> 2014-11-08 18:14:42.672300 7fc5c362f700 5 osd.9 5773 tick -7643> 2014-11-08 18:14:42.879960 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8498 ==== MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3983708842 0 0) 0x1d3af400 con 0x1c5c0580 -7642> 2014-11-08 18:14:42.880003 7fc5a152e700 5 -- op tracker -- seq: 46893, time: 2014-11-08 18:14:42.830021, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7641> 2014-11-08 18:14:42.880016 7fc5a152e700 5 -- op tracker -- seq: 46893, time: 2014-11-08 18:14:42.830026, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7640> 2014-11-08 18:14:42.880024 7fc5a152e700 5 -- op tracker -- seq: 46893, time: 2014-11-08 18:14:42.879952, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7639> 2014-11-08 18:14:42.880032 7fc5a152e700 5 -- op tracker -- seq: 46893, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7638> 2014-11-08 18:14:42.880074 7fc5b0316700 5 -- op tracker -- seq: 46893, time: 2014-11-08 18:14:42.880073, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7637> 2014-11-08 18:14:42.880154 7fc5b0316700 5 -- op tracker -- seq: 46893, time: 2014-11-08 18:14:42.880154, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ff7fcb5/10000019bdc.000000a4/head//4, version: 397'1365, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ff7fcb5/10000019bdc.000000a4/head//4@397'1365, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7636> 2014-11-08 18:14:42.939651 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22d84ec0 con 0x1c5c0580 -7635> 2014-11-08 18:14:42.939784 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(7ff7fcb5/10000019bdc.000000a4/head//4)]) v2 -- ?+0 0x1d3a8000 con 0x1c5c0580 -7634> 2014-11-08 18:14:42.943393 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8499 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) v3 ==== 757+0+0 (598528869 0 0) 0x1d33e300 con 0x1c5c0580 -7633> 2014-11-08 18:14:42.943413 7fc5a152e700 5 -- op tracker -- seq: 46894, time: 2014-11-08 18:14:42.943341, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7632> 2014-11-08 18:14:42.943427 7fc5a152e700 5 -- op tracker -- seq: 46894, time: 2014-11-08 18:14:42.943344, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7631> 2014-11-08 18:14:42.943438 7fc5a152e700 5 -- op tracker -- seq: 46894, time: 2014-11-08 18:14:42.943380, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7630> 2014-11-08 18:14:42.943447 7fc5a152e700 5 -- op tracker -- seq: 46894, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7629> 2014-11-08 18:14:42.943499 7fc5adb11700 5 -- op tracker -- seq: 46894, time: 2014-11-08 18:14:42.943499, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7628> 2014-11-08 18:14:42.943509 7fc5adb11700 5 -- op tracker -- seq: 46894, time: 2014-11-08 18:14:42.943509, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7627> 2014-11-08 18:14:42.943561 7fc5a152e700 5 -- op tracker -- seq: 46894, time: 2014-11-08 18:14:42.943561, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 7ff7fcb5/10000019bdc.000000a4/head//4) -7626> 2014-11-08 18:14:43.042195 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -7625> 2014-11-08 18:14:43.042230 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x211e5a00 con 0x1cae2940 -7624> 2014-11-08 18:14:43.042258 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x211e3840 con 0x1cae2680 -7623> 2014-11-08 18:14:43.042268 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1c831fe0 con 0x1c5c02c0 -7622> 2014-11-08 18:14:43.042277 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d6330c0 con 0x1cae0160 -7621> 2014-11-08 18:14:43.042286 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d630780 con 0x1cae0f20 -7620> 2014-11-08 18:14:43.042295 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d635dc0 con 0x1cae0c60 -7619> 2014-11-08 18:14:43.042304 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x2336f440 con 0x1cae23c0 -7618> 2014-11-08 18:14:43.042313 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x21227440 con 0x1cae56a0 -7617> 2014-11-08 18:14:43.042321 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x2336c740 con 0x1cbf9ce0 -7616> 2014-11-08 18:14:43.042330 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d6312c0 con 0x1d603020 -7615> 2014-11-08 18:14:43.042339 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x233692c0 con 0x1dbd9b80 -7614> 2014-11-08 18:14:43.042347 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d633480 con 0x1d235d80 -7613> 2014-11-08 18:14:43.042356 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d632580 con 0x1cae1fa0 -7612> 2014-11-08 18:14:43.042365 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1d636180 con 0x1cae1ce0 -7611> 2014-11-08 18:14:43.042373 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1caad280 con 0x1cae1340 -7610> 2014-11-08 18:14:43.042382 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1caada00 con 0x1cae1080 -7609> 2014-11-08 18:14:43.042390 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1caac740 con 0x1cae1a20 -7608> 2014-11-08 18:14:43.042399 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:43.042226) v2 -- ?+0 0x1caad460 con 0x1cae2c00 -7607> 2014-11-08 18:14:43.043024 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1d4f2940 con 0x1cae2940 -7606> 2014-11-08 18:14:43.043052 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1f4c2b20 con 0x1cae2680 -7605> 2014-11-08 18:14:43.043082 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x2583d0a0 con 0x1cae56a0 -7604> 2014-11-08 18:14:43.043188 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x2495e180 con 0x1cae23c0 -7603> 2014-11-08 18:14:43.043202 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x257c30c0 con 0x1cae0f20 -7602> 2014-11-08 18:14:43.043353 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1e894b00 con 0x1cae0c60 -7601> 2014-11-08 18:14:43.043440 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x227b2b20 con 0x1cae1fa0 -7600> 2014-11-08 18:14:43.043554 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x257c4740 con 0x1c5c02c0 -7599> 2014-11-08 18:14:43.043567 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x22908780 con 0x1cae0160 -7598> 2014-11-08 18:14:43.043576 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x4535fa0 con 0x1cae1ce0 -7597> 2014-11-08 18:14:43.043585 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 498 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x246caee0 con 0x1cbf9ce0 -7596> 2014-11-08 18:14:43.043595 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 498 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1dbfdfa0 con 0x1d603020 -7595> 2014-11-08 18:14:43.044513 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 305 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1db1dbe0 con 0x1dbd9b80 -7594> 2014-11-08 18:14:43.044529 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 305 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x22d87bc0 con 0x1d235d80 -7593> 2014-11-08 18:14:43.044539 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x25e821c0 con 0x1cae1340 -7592> 2014-11-08 18:14:43.044548 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1decd280 con 0x1cae1080 -7591> 2014-11-08 18:14:43.044556 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1d3201e0 con 0x1cae1a20 -7590> 2014-11-08 18:14:43.044565 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3718 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:43.042226) v2 ==== 47+0+0 (2419708859 0 0) 0x1dbfb480 con 0x1cae2c00 -7589> 2014-11-08 18:14:43.203620 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8500 ==== MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (697428001 0 0) 0x1d3a8000 con 0x1c5c0580 -7588> 2014-11-08 18:14:43.203662 7fc5a152e700 5 -- op tracker -- seq: 46895, time: 2014-11-08 18:14:43.103854, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7587> 2014-11-08 18:14:43.203676 7fc5a152e700 5 -- op tracker -- seq: 46895, time: 2014-11-08 18:14:43.103857, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7586> 2014-11-08 18:14:43.203689 7fc5a152e700 5 -- op tracker -- seq: 46895, time: 2014-11-08 18:14:43.203610, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7585> 2014-11-08 18:14:43.203697 7fc5a152e700 5 -- op tracker -- seq: 46895, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7584> 2014-11-08 18:14:43.203740 7fc5b0316700 5 -- op tracker -- seq: 46895, time: 2014-11-08 18:14:43.203740, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7583> 2014-11-08 18:14:43.203822 7fc5b0316700 5 -- op tracker -- seq: 46895, time: 2014-11-08 18:14:43.203822, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(3818fcb5/1000001a162.000002fd/head//4, version: 397'1890, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3818fcb5/1000001a162.000002fd/head//4@397'1890, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7582> 2014-11-08 18:14:43.277010 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22d82d00 con 0x1c5c0580 -7581> 2014-11-08 18:14:43.277051 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(3818fcb5/1000001a162.000002fd/head//4)]) v2 -- ?+0 0x1d3af400 con 0x1c5c0580 -7580> 2014-11-08 18:14:43.606772 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8501 ==== MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cbbf5b16/ 1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490505+0+0 (1127183662 0 0) 0x229bc200 con 0x1c5c0580 -7579> 2014-11-08 18:14:43.606829 7fc5a152e700 5 -- op tracker -- seq: 46896, time: 2014-11-08 18:14:43.262489, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( cbbf5b16/1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7578> 2014-11-08 18:14:43.606854 7fc5a152e700 5 -- op tracker -- seq: 46896, time: 2014-11-08 18:14:43.262493, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cbbf5b16/ 1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7577> 2014-11-08 18:14:43.606873 7fc5a152e700 5 -- op tracker -- seq: 46896, time: 2014-11-08 18:14:43.606759, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cbbf5b16/ 1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7576> 2014-11-08 18:14:43.606893 7fc5a152e700 5 -- op tracker -- seq: 46896, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cbbf5b16/ 1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7575> 2014-11-08 18:14:43.606966 7fc5ad310700 5 -- op tracker -- seq: 46896, time: 2014-11-08 18:14:43.606966, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( cbbf5b16/1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7574> 2014-11-08 18:14:43.607013 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8502 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) v3 ==== 757+0+0 (4000921894 0 0) 0x22760000 con 0x1c5c0580 -7573> 2014-11-08 18:14:43.607022 7fc5a152e700 5 -- op tracker -- seq: 46897, time: 2014-11-08 18:14:43.606980, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7572> 2014-11-08 18:14:43.607026 7fc5a152e700 5 -- op tracker -- seq: 46897, time: 2014-11-08 18:14:43.606982, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7571> 2014-11-08 18:14:43.607029 7fc5a152e700 5 -- op tracker -- seq: 46897, time: 2014-11-08 18:14:43.607010, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7570> 2014-11-08 18:14:43.607032 7fc5a152e700 5 -- op tracker -- seq: 46897, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7569> 2014-11-08 18:14:43.607146 7fc5ad310700 5 -- op tracker -- seq: 46896, time: 2014-11-08 18:14:43.607146, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(b86e5b16/1000001a467.00000054/head//4, version: 397'1923, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b86e5b16/1000001a467.00000054/head//4@397'1923, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d38e5b16/1000002a69d.0000002c/head//4, version: 3458'5199, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d38e5b16/1000002a69d.0000002c/head//4@3458'5199, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(360f5b16/1000001659e.00000000/head//4, version: 397'553, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(360f5b16/1000001659e.00000000/head//4@397'553, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cbbf5b16/1000002a54f.0000008a/head//4, version: 3033'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cbbf5b16/ 1000002a54f.0000008a/head//4@3033'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3ef5b16/1000002961a.000000b0/head//4, version: 3033'4203, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3ef5b16/1000002961a.000000b0/head//4@3033'4203, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7568> 2014-11-08 18:14:43.607186 7fc5adb11700 5 -- op tracker -- seq: 46897, time: 2014-11-08 18:14:43.607186, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7567> 2014-11-08 18:14:43.607193 7fc5adb11700 5 -- op tracker -- seq: 46897, time: 2014-11-08 18:14:43.607192, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7566> 2014-11-08 18:14:43.607226 7fc5adb11700 5 -- op tracker -- seq: 46897, time: 2014-11-08 18:14:43.607226, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 3818fcb5/1000001a162.000002fd/head//4) -7565> 2014-11-08 18:14:43.639411 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8503 ==== MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2276801288 0 0) 0x1d3af400 con 0x1c5c0580 -7564> 2014-11-08 18:14:43.639460 7fc5a152e700 5 -- op tracker -- seq: 46898, time: 2014-11-08 18:14:43.607047, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7563> 2014-11-08 18:14:43.639478 7fc5a152e700 5 -- op tracker -- seq: 46898, time: 2014-11-08 18:14:43.607048, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7562> 2014-11-08 18:14:43.639523 7fc5a152e700 5 -- op tracker -- seq: 46898, time: 2014-11-08 18:14:43.639386, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7561> 2014-11-08 18:14:43.639536 7fc5a152e700 5 -- op tracker -- seq: 46898, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7560> 2014-11-08 18:14:43.639887 7fc5b0316700 5 -- op tracker -- seq: 46898, time: 2014-11-08 18:14:43.639887, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7559> 2014-11-08 18:14:43.639964 7fc5b0316700 5 -- op tracker -- seq: 46898, time: 2014-11-08 18:14:43.639964, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(f918fcb5/10000019b76.000001e4/head//4, version: 397'1299, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f918fcb5/10000019b76.000001e4/head//4@397'1299, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7558> 2014-11-08 18:14:43.672411 7fc5c362f700 5 osd.9 5773 tick -7557> 2014-11-08 18:14:43.816782 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d87080 con 0x1c5c0580 -7556> 2014-11-08 18:14:43.816821 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d80000 con 0x1c5c0580 -7555> 2014-11-08 18:14:43.816870 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d80d20 con 0x1c5c0580 -7554> 2014-11-08 18:14:43.816913 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d810e0 con 0x1c5c0580 -7553> 2014-11-08 18:14:43.816922 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d83480 con 0x1c5c0580 -7552> 2014-11-08 18:14:43.816974 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(b86e5b16/1000001a467.00000054/head//4),PushReplyOp(d38e5b16/1000002a69d.0000002c/head//4),PushReplyOp(360f5b16/1000001659e.00000000/head//4),PushReplyOp(cbbf5b16/1000002a54f.0000008a/head//4),PushReplyOp(d3ef5b16/1000002961a.000000b0/head//4)]) v2 -- ?+0 0x229b9000 con 0x1c5c0580 -7551> 2014-11-08 18:14:43.823853 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8504 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) v3 ==== 757+0+0 (1503767212 0 0) 0x1d3c9f80 con 0x1c5c0580 -7550> 2014-11-08 18:14:43.823893 7fc5a152e700 5 -- op tracker -- seq: 46899, time: 2014-11-08 18:14:43.823735, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7549> 2014-11-08 18:14:43.823908 7fc5a152e700 5 -- op tracker -- seq: 46899, time: 2014-11-08 18:14:43.823738, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7548> 2014-11-08 18:14:43.823917 7fc5a152e700 5 -- op tracker -- seq: 46899, time: 2014-11-08 18:14:43.823831, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7547> 2014-11-08 18:14:43.823927 7fc5a152e700 5 -- op tracker -- seq: 46899, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7546> 2014-11-08 18:14:43.823970 7fc5afb15700 5 -- op tracker -- seq: 46899, time: 2014-11-08 18:14:43.823970, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7545> 2014-11-08 18:14:43.823983 7fc5afb15700 5 -- op tracker -- seq: 46899, time: 2014-11-08 18:14:43.823983, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7544> 2014-11-08 18:14:43.824032 7fc5afb15700 5 -- op tracker -- seq: 46899, time: 2014-11-08 18:14:43.824032, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb d3ef5b16/1000002961a.000000b0/head//4) -7543> 2014-11-08 18:14:43.895451 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22d82940 con 0x1c5c0580 -7542> 2014-11-08 18:14:43.895506 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(f918fcb5/10000019b76.000001e4/head//4)]) v2 -- ?+0 0x1d3a8000 con 0x1c5c0580 -7541> 2014-11-08 18:14:43.915330 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8505 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) v3 ==== 757+0+0 (4108222956 0 0) 0x22763f00 con 0x1c5c0580 -7540> 2014-11-08 18:14:43.915385 7fc5a152e700 5 -- op tracker -- seq: 46900, time: 2014-11-08 18:14:43.915201, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7539> 2014-11-08 18:14:43.915393 7fc5a152e700 5 -- op tracker -- seq: 46900, time: 2014-11-08 18:14:43.915203, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7538> 2014-11-08 18:14:43.915396 7fc5a152e700 5 -- op tracker -- seq: 46900, time: 2014-11-08 18:14:43.915321, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7537> 2014-11-08 18:14:43.915399 7fc5a152e700 5 -- op tracker -- seq: 46900, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7536> 2014-11-08 18:14:43.915439 7fc5adb11700 5 -- op tracker -- seq: 46900, time: 2014-11-08 18:14:43.915439, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7535> 2014-11-08 18:14:43.915452 7fc5adb11700 5 -- op tracker -- seq: 46900, time: 2014-11-08 18:14:43.915452, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7534> 2014-11-08 18:14:43.915525 7fc5adb11700 5 -- op tracker -- seq: 46900, time: 2014-11-08 18:14:43.915525, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f918fcb5/10000019b76.000001e4/head//4) -7533> 2014-11-08 18:14:44.072838 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12046 ==== MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 298+0+0 (570265679 0 0) 0x1ce64c80 con 0x1c5c2680 -7532> 2014-11-08 18:14:44.072886 7fc5a6cfe700 5 -- op tracker -- seq: 46901, time: 2014-11-08 18:14:44.072721, event: header_read, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) -7531> 2014-11-08 18:14:44.072898 7fc5a6cfe700 5 -- op tracker -- seq: 46901, time: 2014-11-08 18:14:44.072728, event: throttled, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) -7530> 2014-11-08 18:14:44.072903 7fc5a6cfe700 5 -- op tracker -- seq: 46901, time: 2014-11-08 18:14:44.072823, event: all_read, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) -7529> 2014-11-08 18:14:44.072908 7fc5a6cfe700 5 -- op tracker -- seq: 46901, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) -7528> 2014-11-08 18:14:44.072999 7fc5ad310700 5 -- op tracker -- seq: 46901, time: 2014-11-08 18:14:44.072999, event: reached_pg, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) -7527> 2014-11-08 18:14:44.087786 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12047 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (1133990783 0 0) 0x1ce64140 con 0x1c5c2680 -7526> 2014-11-08 18:14:44.087811 7fc5a6cfe700 5 -- op tracker -- seq: 46902, time: 2014-11-08 18:14:44.087685, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) -7525> 2014-11-08 18:14:44.087819 7fc5a6cfe700 5 -- op tracker -- seq: 46902, time: 2014-11-08 18:14:44.087693, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) -7524> 2014-11-08 18:14:44.087824 7fc5a6cfe700 5 -- op tracker -- seq: 46902, time: 2014-11-08 18:14:44.087779, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) -7523> 2014-11-08 18:14:44.087829 7fc5a6cfe700 5 -- op tracker -- seq: 46902, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) -7522> 2014-11-08 18:14:44.087866 7fc5aeb13700 5 -- op tracker -- seq: 46902, time: 2014-11-08 18:14:44.087866, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) -7521> 2014-11-08 18:14:44.094019 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8506 ==== MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1909106085 0 0) 0x1d3a8000 con 0x1c5c0580 -7520> 2014-11-08 18:14:44.094048 7fc5a152e700 5 -- op tracker -- seq: 46903, time: 2014-11-08 18:14:44.052195, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7519> 2014-11-08 18:14:44.094059 7fc5a152e700 5 -- op tracker -- seq: 46903, time: 2014-11-08 18:14:44.052200, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7518> 2014-11-08 18:14:44.094067 7fc5a152e700 5 -- op tracker -- seq: 46903, time: 2014-11-08 18:14:44.093978, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7517> 2014-11-08 18:14:44.094075 7fc5a152e700 5 -- op tracker -- seq: 46903, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7516> 2014-11-08 18:14:44.094128 7fc5b0316700 5 -- op tracker -- seq: 46903, time: 2014-11-08 18:14:44.094128, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7515> 2014-11-08 18:14:44.094149 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8507 ==== MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (12422003 0 0) 0x224af740 con 0x1c5c0580 -7514> 2014-11-08 18:14:44.094162 7fc5a152e700 5 -- op tracker -- seq: 46904, time: 2014-11-08 18:14:44.094108, event: header_read, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) -7513> 2014-11-08 18:14:44.094168 7fc5a152e700 5 -- op tracker -- seq: 46904, time: 2014-11-08 18:14:44.094108, event: throttled, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) -7512> 2014-11-08 18:14:44.094174 7fc5a152e700 5 -- op tracker -- seq: 46904, time: 2014-11-08 18:14:44.094140, event: all_read, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) -7511> 2014-11-08 18:14:44.094179 7fc5a152e700 5 -- op tracker -- seq: 46904, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) -7510> 2014-11-08 18:14:44.094205 7fc5acb0f700 5 -- op tracker -- seq: 46904, time: 2014-11-08 18:14:44.094205, event: reached_pg, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) -7509> 2014-11-08 18:14:44.094206 7fc5b0316700 5 -- op tracker -- seq: 46903, time: 2014-11-08 18:14:44.094206, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(fd88fcb5/1000001ac54.00000019/head//4, version: 2829'2640, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fd88fcb5/1000001ac54.00000019/head//4@2829'2640, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7508> 2014-11-08 18:14:44.186064 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22d83660 con 0x1c5c0580 -7507> 2014-11-08 18:14:44.186110 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(fd88fcb5/1000001ac54.00000019/head//4)]) v2 -- ?+0 0x1d3af400 con 0x1c5c0580 -7506> 2014-11-08 18:14:44.192049 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8508 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) v3 ==== 757+0+0 (504097334 0 0) 0x1d3ce300 con 0x1c5c0580 -7505> 2014-11-08 18:14:44.192074 7fc5a152e700 5 -- op tracker -- seq: 46905, time: 2014-11-08 18:14:44.191970, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7504> 2014-11-08 18:14:44.192085 7fc5a152e700 5 -- op tracker -- seq: 46905, time: 2014-11-08 18:14:44.191974, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7503> 2014-11-08 18:14:44.192089 7fc5a152e700 5 -- op tracker -- seq: 46905, time: 2014-11-08 18:14:44.192043, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7502> 2014-11-08 18:14:44.192093 7fc5a152e700 5 -- op tracker -- seq: 46905, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7501> 2014-11-08 18:14:44.192145 7fc5adb11700 5 -- op tracker -- seq: 46905, time: 2014-11-08 18:14:44.192145, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7500> 2014-11-08 18:14:44.192157 7fc5adb11700 5 -- op tracker -- seq: 46905, time: 2014-11-08 18:14:44.192157, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7499> 2014-11-08 18:14:44.192207 7fc5adb11700 5 -- op tracker -- seq: 46905, time: 2014-11-08 18:14:44.192207, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb fd88fcb5/1000001ac54.00000019/head//4) -7498> 2014-11-08 18:14:44.249633 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1586 ==== MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) v2 ==== 110+0+0 (3401857894 0 0) 0x24847200 con 0x21468c60 -7497> 2014-11-08 18:14:44.249670 7fc5a39d2700 5 -- op tracker -- seq: 46906, time: 2014-11-08 18:14:44.249536, event: header_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) -7496> 2014-11-08 18:14:44.249680 7fc5a39d2700 5 -- op tracker -- seq: 46906, time: 2014-11-08 18:14:44.249541, event: throttled, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) -7495> 2014-11-08 18:14:44.249694 7fc5a39d2700 5 -- op tracker -- seq: 46906, time: 2014-11-08 18:14:44.249625, event: all_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) -7494> 2014-11-08 18:14:44.249698 7fc5a39d2700 5 -- op tracker -- seq: 46906, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) -7493> 2014-11-08 18:14:44.249762 7fc5b0316700 5 -- op tracker -- seq: 46906, time: 2014-11-08 18:14:44.249762, event: reached_pg, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) -7492> 2014-11-08 18:14:44.249845 7fc5b0316700 5 -- op tracker -- seq: 46906, time: 2014-11-08 18:14:44.249844, event: done, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(2bc7de15/1000002bd1a.0000007e/head//4)]) -7491> 2014-11-08 18:14:44.300435 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3671 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:44.298058) v2 ==== 47+0+0 (4291524268 0 0) 0x224a3840 con 0x1cce2ec0 -7490> 2014-11-08 18:14:44.300460 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:44.298058) v2 -- ?+0 0x1f4c5dc0 con 0x1cce2ec0 -7489> 2014-11-08 18:14:44.300501 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3671 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:44.298058) v2 ==== 47+0+0 (4291524268 0 0) 0x23142760 con 0x1cce3440 -7488> 2014-11-08 18:14:44.300512 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:44.298058) v2 -- ?+0 0x2495f800 con 0x1cce3440 -7487> 2014-11-08 18:14:44.644697 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1587 ==== MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) v2 ==== 110+0+0 (1124150144 0 0) 0x1ebb5c00 con 0x21468c60 -7486> 2014-11-08 18:14:44.644735 7fc5a39d2700 5 -- op tracker -- seq: 46907, time: 2014-11-08 18:14:44.644587, event: header_read, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) -7485> 2014-11-08 18:14:44.644746 7fc5a39d2700 5 -- op tracker -- seq: 46907, time: 2014-11-08 18:14:44.644598, event: throttled, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) -7484> 2014-11-08 18:14:44.644750 7fc5a39d2700 5 -- op tracker -- seq: 46907, time: 2014-11-08 18:14:44.644681, event: all_read, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) -7483> 2014-11-08 18:14:44.644754 7fc5a39d2700 5 -- op tracker -- seq: 46907, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) -7482> 2014-11-08 18:14:44.644809 7fc5afb15700 5 -- op tracker -- seq: 46907, time: 2014-11-08 18:14:44.644808, event: reached_pg, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) -7481> 2014-11-08 18:14:44.644882 7fc5afb15700 5 -- op tracker -- seq: 46907, time: 2014-11-08 18:14:44.644882, event: done, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(a2c83e0c/1000002a7e4.00000072/head//4)]) -7480> 2014-11-08 18:14:44.672516 7fc5c362f700 5 osd.9 5773 tick -7479> 2014-11-08 18:14:44.834656 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8509 ==== MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490148+0+0 (1913017983 0 0) 0x1d3af400 con 0x1c5c0580 -7478> 2014-11-08 18:14:44.834735 7fc5a152e700 5 -- op tracker -- seq: 46908, time: 2014-11-08 18:14:44.614532, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7477> 2014-11-08 18:14:44.834758 7fc5a152e700 5 -- op tracker -- seq: 46908, time: 2014-11-08 18:14:44.614537, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7476> 2014-11-08 18:14:44.834775 7fc5a152e700 5 -- op tracker -- seq: 46908, time: 2014-11-08 18:14:44.834620, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7475> 2014-11-08 18:14:44.834793 7fc5a152e700 5 -- op tracker -- seq: 46908, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7474> 2014-11-08 18:14:44.834895 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8510 ==== MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) v1 ==== 219+0+0 (4162516553 0 0) 0x224ada00 con 0x1c5c0580 -7473> 2014-11-08 18:14:44.834905 7fc5a152e700 5 -- op tracker -- seq: 46909, time: 2014-11-08 18:14:44.834844, event: header_read, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) -7472> 2014-11-08 18:14:44.834910 7fc5a152e700 5 -- op tracker -- seq: 46909, time: 2014-11-08 18:14:44.834845, event: throttled, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) -7471> 2014-11-08 18:14:44.834914 7fc5a152e700 5 -- op tracker -- seq: 46909, time: 2014-11-08 18:14:44.834885, event: all_read, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) -7470> 2014-11-08 18:14:44.834918 7fc5a152e700 5 -- op tracker -- seq: 46909, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) -7469> 2014-11-08 18:14:44.836629 7fc5afb15700 5 -- op tracker -- seq: 46908, time: 2014-11-08 18:14:44.836629, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7468> 2014-11-08 18:14:44.836825 7fc5afb15700 5 -- op tracker -- seq: 46908, time: 2014-11-08 18:14:44.836824, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(2f106b16/1000001a246.0000001d/head//4, version: 397'1833, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f106b16/1000001a246.0000001d/head//4@397'1833, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(77406b16/10000019fec.0000040a/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77406b16/10000019fec.0000040a/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(32c06b16/100000199ce.000002ba/head//4, version: 397'836, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32c06b16/100000199ce.000002ba/head//4@397'836, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75e06b16/1000002a416.000000d0/head//4, version: 3033'4902, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e06b16/ 1000002a416.000000d0/head//4@3033'4902, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(25816b16/1000002bdb7.0000008d/head//4, version: 4864'8340, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(25816b16/1000002bdb7.0000008d/head//4@4864'8340, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7467> 2014-11-08 18:14:44.836865 7fc5af314700 5 -- op tracker -- seq: 46909, time: 2014-11-08 18:14:44.836865, event: reached_pg, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) -7466> 2014-11-08 18:14:44.967793 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1588 ==== MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (1180514044 0 0) 0x22878480 con 0x21468c60 -7465> 2014-11-08 18:14:44.967837 7fc5a39d2700 5 -- op tracker -- seq: 46910, time: 2014-11-08 18:14:44.967675, event: header_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) -7464> 2014-11-08 18:14:44.967850 7fc5a39d2700 5 -- op tracker -- seq: 46910, time: 2014-11-08 18:14:44.967680, event: throttled, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) -7463> 2014-11-08 18:14:44.967858 7fc5a39d2700 5 -- op tracker -- seq: 46910, time: 2014-11-08 18:14:44.967785, event: all_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) -7462> 2014-11-08 18:14:44.967865 7fc5a39d2700 5 -- op tracker -- seq: 46910, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) -7461> 2014-11-08 18:14:44.968282 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1589 ==== pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) v3 ==== 757+0+0 (890958727 0 0) 0x1e58b600 con 0x21468c60 -7460> 2014-11-08 18:14:44.968299 7fc5a39d2700 5 -- op tracker -- seq: 46911, time: 2014-11-08 18:14:44.968225, event: header_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7459> 2014-11-08 18:14:44.968307 7fc5a39d2700 5 -- op tracker -- seq: 46911, time: 2014-11-08 18:14:44.968227, event: throttled, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7458> 2014-11-08 18:14:44.968316 7fc5a39d2700 5 -- op tracker -- seq: 46911, time: 2014-11-08 18:14:44.968278, event: all_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7457> 2014-11-08 18:14:44.968321 7fc5a39d2700 5 -- op tracker -- seq: 46911, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7456> 2014-11-08 18:14:45.016075 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpRead(4.15s2 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=4e7de15/1000002b702.00000065/head//4)) v1 -- ?+0 0x224ab180 con 0x1c5c2680 -7455> 2014-11-08 18:14:45.016111 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x224a8fc0 con 0x42c86e0 -7454> 2014-11-08 18:14:45.016128 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x224a8fc0 con 0x42c86e0 -7453> 2014-11-08 18:14:45.016144 7fc5b9328700 5 -- op tracker -- seq: 46912, time: 2014-11-08 18:14:45.016119, event: header_read, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) -7452> 2014-11-08 18:14:45.016152 7fc5b9328700 5 -- op tracker -- seq: 46912, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) -7451> 2014-11-08 18:14:45.016155 7fc5b9328700 5 -- op tracker -- seq: 46912, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) -7450> 2014-11-08 18:14:45.016158 7fc5b9328700 5 -- op tracker -- seq: 46912, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) -7449> 2014-11-08 18:14:45.016124 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=1 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is c958de15/10000028657.00000051/head//4 -7448> 2014-11-08 18:14:45.016176 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_backfill(progress 4.15s0 e 5773/5773 lb 2bc7de15/1000002bd1a.0000007e/head//4) v3 -- ?+0 0x1d3ccc80 con 0x1c5c2680 -7447> 2014-11-08 18:14:45.016186 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_backfill(progress 4.15s2 e 5773/5773 lb 2bc7de15/1000002bd1a.0000007e/head//4) v3 -- ?+0 0x226f3600 con 0x21468c60 -7446> 2014-11-08 18:14:45.016197 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_backfill(progress 4.15s1 e 5773/5773 lb 2bc7de15/1000002bd1a.0000007e/head//4) v3 -- ?+0 0x226f6780 con 0x1c7911e0 -7445> 2014-11-08 18:14:45.016325 7fc5adb11700 5 -- op tracker -- seq: 46912, time: 2014-11-08 18:14:45.016325, event: reached_pg, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) -7444> 2014-11-08 18:14:45.016393 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpRead(4.cs1 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=fe393e0c/1000001b50a.000000f6/head//4)) v1 -- ?+0 0x1bea7500 con 0x1c5c2680 -7443> 2014-11-08 18:14:45.016403 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2302f080 con 0x42c86e0 -7442> 2014-11-08 18:14:45.016412 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=1 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 71c93e0c/1000001a6bb.000000cd/head//4 -7441> 2014-11-08 18:14:45.016427 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_backfill(progress 4.cs2 e 5773/5773 lb a2c83e0c/1000002a7e4.00000072/head//4) v3 -- ?+0 0x226f7980 con 0x21468c60 -7440> 2014-11-08 18:14:45.016602 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2302f080 con 0x42c86e0 -7439> 2014-11-08 18:14:45.016631 7fc5b9328700 5 -- op tracker -- seq: 46913, time: 2014-11-08 18:14:45.016408, event: header_read, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) -7438> 2014-11-08 18:14:45.016635 7fc5b9328700 5 -- op tracker -- seq: 46913, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) -7437> 2014-11-08 18:14:45.016638 7fc5b9328700 5 -- op tracker -- seq: 46913, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) -7436> 2014-11-08 18:14:45.016640 7fc5b9328700 5 -- op tracker -- seq: 46913, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) -7435> 2014-11-08 18:14:45.016676 7fc5afb15700 5 -- op tracker -- seq: 46913, time: 2014-11-08 18:14:45.016676, event: reached_pg, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) -7434> 2014-11-08 18:14:45.036800 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8511 ==== MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3863123167 0 0) 0x229b9000 con 0x1c5c0580 -7433> 2014-11-08 18:14:45.036846 7fc5a152e700 5 -- op tracker -- seq: 46914, time: 2014-11-08 18:14:44.997922, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7432> 2014-11-08 18:14:45.036859 7fc5a152e700 5 -- op tracker -- seq: 46914, time: 2014-11-08 18:14:44.997926, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7431> 2014-11-08 18:14:45.036866 7fc5a152e700 5 -- op tracker -- seq: 46914, time: 2014-11-08 18:14:45.036767, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7430> 2014-11-08 18:14:45.036873 7fc5a152e700 5 -- op tracker -- seq: 46914, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7429> 2014-11-08 18:14:45.036915 7fc5b0316700 5 -- op tracker -- seq: 46914, time: 2014-11-08 18:14:45.036915, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7428> 2014-11-08 18:14:45.036997 7fc5b0316700 5 -- op tracker -- seq: 46914, time: 2014-11-08 18:14:45.036997, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(309fcb5/1000001ac25.00000051/head//4, version: 2829'2631, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(309fcb5/1000001ac25.00000051/head//4@2829'2631, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7427> 2014-11-08 18:14:45.108077 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1590 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2972446705 0 0) 0x22878240 con 0x21468c60 -7426> 2014-11-08 18:14:45.108106 7fc5a39d2700 5 -- op tracker -- seq: 46915, time: 2014-11-08 18:14:45.107984, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) -7425> 2014-11-08 18:14:45.108131 7fc5a39d2700 5 -- op tracker -- seq: 46915, time: 2014-11-08 18:14:45.107986, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) -7424> 2014-11-08 18:14:45.108134 7fc5a39d2700 5 -- op tracker -- seq: 46915, time: 2014-11-08 18:14:45.108070, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) -7423> 2014-11-08 18:14:45.108137 7fc5a39d2700 5 -- op tracker -- seq: 46915, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) -7422> 2014-11-08 18:14:45.170395 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12048 ==== MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) v1 ==== 2097696+0+0 (3858828593 0 0) 0x211d6e00 con 0x1c5c2680 -7421> 2014-11-08 18:14:45.170434 7fc5a6cfe700 5 -- op tracker -- seq: 46916, time: 2014-11-08 18:14:45.149518, event: header_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) -7420> 2014-11-08 18:14:45.170462 7fc5a6cfe700 5 -- op tracker -- seq: 46916, time: 2014-11-08 18:14:45.149521, event: throttled, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) -7419> 2014-11-08 18:14:45.170472 7fc5a6cfe700 5 -- op tracker -- seq: 46916, time: 2014-11-08 18:14:45.170370, event: all_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) -7418> 2014-11-08 18:14:45.170480 7fc5a6cfe700 5 -- op tracker -- seq: 46916, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) -7417> 2014-11-08 18:14:45.254454 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22d821c0 con 0x1c5c0580 -7416> 2014-11-08 18:14:45.254492 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c5280 con 0x1c5c0580 -7415> 2014-11-08 18:14:45.254615 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c23a0 con 0x1c5c0580 -7414> 2014-11-08 18:14:45.254622 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x233fd640 con 0x1c5c0580 -7413> 2014-11-08 18:14:45.254627 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x233fc920 con 0x1c5c0580 -7412> 2014-11-08 18:14:45.254635 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(2f106b16/1000001a246.0000001d/head//4),PushReplyOp(77406b16/10000019fec.0000040a/head//4),PushReplyOp(32c06b16/100000199ce.000002ba/head//4),PushReplyOp(75e06b16/1000002a416.000000d0/head//4),PushReplyOp(25816b16/1000002bdb7.0000008d/head//4)]) v2 -- ?+0 0x1ebb5c00 con 0x1c5c0580 -7411> 2014-11-08 18:14:45.259895 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8512 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) v3 ==== 757+0+0 (525259353 0 0) 0x1e588000 con 0x1c5c0580 -7410> 2014-11-08 18:14:45.259935 7fc5a152e700 5 -- op tracker -- seq: 46917, time: 2014-11-08 18:14:45.259636, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7409> 2014-11-08 18:14:45.259950 7fc5a152e700 5 -- op tracker -- seq: 46917, time: 2014-11-08 18:14:45.259639, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7408> 2014-11-08 18:14:45.260025 7fc5a152e700 5 -- op tracker -- seq: 46917, time: 2014-11-08 18:14:45.259874, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7407> 2014-11-08 18:14:45.260034 7fc5a152e700 5 -- op tracker -- seq: 46917, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7406> 2014-11-08 18:14:45.325037 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) v1 -- ?+0 0x224beb80 con 0x42c86e0 -7405> 2014-11-08 18:14:45.325084 7fc5adb11700 5 -- op tracker -- seq: 46912, time: 2014-11-08 18:14:45.325084, event: done, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4406, to_read={4e7de15/1000002b702.00000065/head//4=0,4194304}, attrs_to_read=)) -7404> 2014-11-08 18:14:45.325123 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224beb80 con 0x42c86e0 -7403> 2014-11-08 18:14:45.325136 7fc5b9328700 5 -- op tracker -- seq: 46918, time: 2014-11-08 18:14:45.325074, event: header_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) -7402> 2014-11-08 18:14:45.325139 7fc5b9328700 5 -- op tracker -- seq: 46918, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) -7401> 2014-11-08 18:14:45.325141 7fc5b9328700 5 -- op tracker -- seq: 46918, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) -7400> 2014-11-08 18:14:45.325143 7fc5b9328700 5 -- op tracker -- seq: 46918, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) -7399> 2014-11-08 18:14:45.325175 7fc5b0316700 5 -- op tracker -- seq: 46918, time: 2014-11-08 18:14:45.325175, event: reached_pg, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) -7398> 2014-11-08 18:14:45.325217 7fc5b0316700 5 -- op tracker -- seq: 46918, time: 2014-11-08 18:14:45.325216, event: done, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=0)) -7397> 2014-11-08 18:14:45.342845 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist [0,0,0,0,0,0,0,7,2,1]) -7396> 2014-11-08 18:14:45.342878 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x21226ae0 con 0x1cae2940 -7395> 2014-11-08 18:14:45.342904 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x1caa9fe0 con 0x1cae2680 -7394> 2014-11-08 18:14:45.342915 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x1caa92c0 con 0x1c5c02c0 -7393> 2014-11-08 18:14:45.342926 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x1d630000 con 0x1cae0160 -7392> 2014-11-08 18:14:45.342936 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x1d637080 con 0x1cae0f20 -7391> 2014-11-08 18:14:45.342947 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x1d631860 con 0x1cae0c60 -7390> 2014-11-08 18:14:45.342957 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x211e2b20 con 0x1cae23c0 -7389> 2014-11-08 18:14:45.342967 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x211e5820 con 0x1cae56a0 -7388> 2014-11-08 18:14:45.342977 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x211e3a20 con 0x1cbf9ce0 -7387> 2014-11-08 18:14:45.342988 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x1e534b00 con 0x1d603020 -7386> 2014-11-08 18:14:45.342998 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x2515ba20 con 0x1dbd9b80 -7385> 2014-11-08 18:14:45.343008 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x2515f080 con 0x1d235d80 -7384> 2014-11-08 18:14:45.343018 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x25ed1a40 con 0x1cae1fa0 -7383> 2014-11-08 18:14:45.343029 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x2515ecc0 con 0x1cae1ce0 -7382> 2014-11-08 18:14:45.343039 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x25fc2580 con 0x1cae1340 -7381> 2014-11-08 18:14:45.343049 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x25fc4ce0 con 0x1cae1080 -7380> 2014-11-08 18:14:45.343059 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x2286f080 con 0x1cae1a20 -7379> 2014-11-08 18:14:45.343068 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:45.342873) v2 -- ?+0 0x258383c0 con 0x1cae2c00 -7378> 2014-11-08 18:14:45.344029 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x23687800 con 0x1cae2940 -7377> 2014-11-08 18:14:45.344050 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1f4c6180 con 0x1cae2680 -7376> 2014-11-08 18:14:45.344061 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x24959c20 con 0x1cae23c0 -7375> 2014-11-08 18:14:45.344073 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x2583ecc0 con 0x1cae56a0 -7374> 2014-11-08 18:14:45.344083 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 306 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1db1e720 con 0x1dbd9b80 -7373> 2014-11-08 18:14:45.344095 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 306 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x22d812c0 con 0x1d235d80 -7372> 2014-11-08 18:14:45.344106 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1eacc560 con 0x1cae1fa0 -7371> 2014-11-08 18:14:45.344249 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1e8912c0 con 0x1cae0c60 -7370> 2014-11-08 18:14:45.344262 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x4530000 con 0x1cae1ce0 -7369> 2014-11-08 18:14:45.344274 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 499 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x246ce720 con 0x1cbf9ce0 -7368> 2014-11-08 18:14:45.344338 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 499 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x22349680 con 0x1d603020 -7367> 2014-11-08 18:14:45.344350 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x257c3a20 con 0x1c5c02c0 -7366> 2014-11-08 18:14:45.344361 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x229090e0 con 0x1cae0160 -7365> 2014-11-08 18:14:45.344372 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1e533660 con 0x1cae0f20 -7364> 2014-11-08 18:14:45.345003 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1deccb00 con 0x1cae1080 -7363> 2014-11-08 18:14:45.345023 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x25e84920 con 0x1cae1340 -7362> 2014-11-08 18:14:45.345549 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1dbfcb00 con 0x1cae2c00 -7361> 2014-11-08 18:14:45.345580 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3719 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.342873) v2 ==== 47+0+0 (843336199 0 0) 0x1d3205a0 con 0x1cae1a20 -7360> 2014-11-08 18:14:45.377056 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) v1 -- ?+0 0x211d1180 con 0x42c86e0 -7359> 2014-11-08 18:14:45.377101 7fc5afb15700 5 -- op tracker -- seq: 46913, time: 2014-11-08 18:14:45.377100, event: done, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4407, to_read={fe393e0c/1000001b50a.000000f6/head//4=0,4194304}, attrs_to_read=)) -7358> 2014-11-08 18:14:45.377132 7fc5afb15700 5 -- op tracker -- seq: 46916, time: 2014-11-08 18:14:45.377132, event: reached_pg, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) -7357> 2014-11-08 18:14:45.377166 7fc5afb15700 5 -- op tracker -- seq: 46916, time: 2014-11-08 18:14:45.377166, event: done, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=1)) -7356> 2014-11-08 18:14:45.377177 7fc5afb15700 5 -- op tracker -- seq: 46917, time: 2014-11-08 18:14:45.377177, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7355> 2014-11-08 18:14:45.377184 7fc5afb15700 5 -- op tracker -- seq: 46917, time: 2014-11-08 18:14:45.377184, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7354> 2014-11-08 18:14:45.377218 7fc5afb15700 5 -- op tracker -- seq: 46917, time: 2014-11-08 18:14:45.377218, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 25816b16/1000002bdb7.0000008d/head//4) -7353> 2014-11-08 18:14:45.377242 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22457260 con 0x1c5c0580 -7352> 2014-11-08 18:14:45.377257 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(309fcb5/1000001ac25.00000051/head//4)]) v2 -- ?+0 0x24847200 con 0x1c5c0580 -7351> 2014-11-08 18:14:45.377397 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d1180 con 0x42c86e0 -7350> 2014-11-08 18:14:45.377419 7fc5b9328700 5 -- op tracker -- seq: 46919, time: 2014-11-08 18:14:45.377092, event: header_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) -7349> 2014-11-08 18:14:45.377422 7fc5b9328700 5 -- op tracker -- seq: 46919, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) -7348> 2014-11-08 18:14:45.377424 7fc5b9328700 5 -- op tracker -- seq: 46919, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) -7347> 2014-11-08 18:14:45.377426 7fc5b9328700 5 -- op tracker -- seq: 46919, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) -7346> 2014-11-08 18:14:45.377608 7fc5afb15700 5 -- op tracker -- seq: 46919, time: 2014-11-08 18:14:45.377608, event: reached_pg, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) -7345> 2014-11-08 18:14:45.382568 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPush(4.cs2 5773 [PushOp(fe393e0c/1000001b50a.000000f6/head//4, version: 3033'3083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe393e0c/1000001b50a.000000f6/head//4@3033'3083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3af400 con 0x21468c60 -7344> 2014-11-08 18:14:45.382665 7fc5afb15700 5 -- op tracker -- seq: 46919, time: 2014-11-08 18:14:45.382665, event: done, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4407, attrs_read=0)) -7343> 2014-11-08 18:14:45.393092 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8513 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) v3 ==== 757+0+0 (2606773471 0 0) 0x1ce9ba80 con 0x1c5c0580 -7342> 2014-11-08 18:14:45.393132 7fc5a152e700 5 -- op tracker -- seq: 46920, time: 2014-11-08 18:14:45.392988, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7341> 2014-11-08 18:14:45.393146 7fc5a152e700 5 -- op tracker -- seq: 46920, time: 2014-11-08 18:14:45.392992, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7340> 2014-11-08 18:14:45.393156 7fc5a152e700 5 -- op tracker -- seq: 46920, time: 2014-11-08 18:14:45.393069, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7339> 2014-11-08 18:14:45.393165 7fc5a152e700 5 -- op tracker -- seq: 46920, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7338> 2014-11-08 18:14:45.393206 7fc5adb11700 5 -- op tracker -- seq: 46920, time: 2014-11-08 18:14:45.393206, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7337> 2014-11-08 18:14:45.393216 7fc5adb11700 5 -- op tracker -- seq: 46920, time: 2014-11-08 18:14:45.393216, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7336> 2014-11-08 18:14:45.393383 7fc5a152e700 5 -- op tracker -- seq: 46920, time: 2014-11-08 18:14:45.393383, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 309fcb5/1000001ac25.00000051/head//4) -7335> 2014-11-08 18:14:45.402864 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25186, attrs_read=0)) v1 -- ?+0 0x1c89bc00 con 0x1c5c2680 -7334> 2014-11-08 18:14:45.402919 7fc5aeb13700 5 -- op tracker -- seq: 46902, time: 2014-11-08 18:14:45.402919, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25186, to_read={504c3227/100000199e7.000003b5/head//4=0,4194304}, attrs_to_read=)) -7333> 2014-11-08 18:14:45.403755 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3672 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:45.398703) v2 ==== 47+0+0 (1817133287 0 0) 0x23146900 con 0x1cce3440 -7332> 2014-11-08 18:14:45.403775 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.398703) v2 -- ?+0 0x23142760 con 0x1cce3440 -7331> 2014-11-08 18:14:45.403801 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3672 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:45.398703) v2 ==== 47+0+0 (1817133287 0 0) 0x224a23a0 con 0x1cce2ec0 -7330> 2014-11-08 18:14:45.403812 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.398703) v2 -- ?+0 0x224a3840 con 0x1cce2ec0 -7329> 2014-11-08 18:14:45.570397 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1591 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1eae5280 con 0x21468c60 -7328> 2014-11-08 18:14:45.570429 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570308, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -7327> 2014-11-08 18:14:45.570439 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570311, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -7326> 2014-11-08 18:14:45.570442 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570374, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -7325> 2014-11-08 18:14:45.570446 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570424, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -7324> 2014-11-08 18:14:45.570450 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570450, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -7323> 2014-11-08 18:14:45.570460 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570459, event: started, op: pg_trim(4.c to 5734'8057 e5773) -7322> 2014-11-08 18:14:45.570467 7fc5b9b29700 5 -- op tracker -- seq: 46921, time: 2014-11-08 18:14:45.570467, event: done, op: pg_trim(4.c to 5734'8057 e5773) -7321> 2014-11-08 18:14:45.611006 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpReadReply(4.bcs0 5773 ECSubReadReply(tid=20431, attrs_read=0)) v1 -- ?+0 0x1c898c80 con 0x1c5c0580 -7320> 2014-11-08 18:14:45.611048 7fc5af314700 5 -- op tracker -- seq: 46909, time: 2014-11-08 18:14:45.611047, event: done, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20431, to_read={e11517bc/1000002bb26.00000077/head//4=0,4194304,5e5517bc/10000019b45.000009fb/head//4=0,4194304}, attrs_to_read=)) -7319> 2014-11-08 18:14:45.611081 7fc5af314700 5 -- op tracker -- seq: 46911, time: 2014-11-08 18:14:45.611081, event: reached_pg, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7318> 2014-11-08 18:14:45.611092 7fc5af314700 5 -- op tracker -- seq: 46911, time: 2014-11-08 18:14:45.611092, event: started, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7317> 2014-11-08 18:14:45.611144 7fc5af314700 5 -- op tracker -- seq: 46911, time: 2014-11-08 18:14:45.611144, event: done, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 76aa678f/1000002a47e.00000015/head//4) -7316> 2014-11-08 18:14:45.611159 7fc5af314700 5 -- op tracker -- seq: 46910, time: 2014-11-08 18:14:45.611159, event: reached_pg, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) -7315> 2014-11-08 18:14:45.614158 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2212 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) v3 ==== 757+0+0 (2347655624 0 0) 0x1d3ca880 con 0x1e50aec0 -7314> 2014-11-08 18:14:45.614194 7fc5a56ef700 5 -- op tracker -- seq: 46922, time: 2014-11-08 18:14:45.614088, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7313> 2014-11-08 18:14:45.614200 7fc5a56ef700 5 -- op tracker -- seq: 46922, time: 2014-11-08 18:14:45.614091, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7312> 2014-11-08 18:14:45.614203 7fc5a56ef700 5 -- op tracker -- seq: 46922, time: 2014-11-08 18:14:45.614150, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7311> 2014-11-08 18:14:45.614206 7fc5a56ef700 5 -- op tracker -- seq: 46922, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7310> 2014-11-08 18:14:45.672666 7fc5c362f700 5 osd.9 5773 tick -7309> 2014-11-08 18:14:45.672716 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -7308> 2014-11-08 18:14:45.672724 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(2 pgs tid 1327 v 0) v1 -- ?+0 0x248aac00 con 0x44ed6a0 -7307> 2014-11-08 18:14:45.693034 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12049 ==== MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) v1 ==== 2097696+0+0 (942897581 0 0) 0x1c89bc00 con 0x1c5c2680 -7306> 2014-11-08 18:14:45.693064 7fc5a6cfe700 5 -- op tracker -- seq: 46923, time: 2014-11-08 18:14:45.640323, event: header_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) -7305> 2014-11-08 18:14:45.693072 7fc5a6cfe700 5 -- op tracker -- seq: 46923, time: 2014-11-08 18:14:45.640325, event: throttled, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) -7304> 2014-11-08 18:14:45.693075 7fc5a6cfe700 5 -- op tracker -- seq: 46923, time: 2014-11-08 18:14:45.693026, event: all_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) -7303> 2014-11-08 18:14:45.693078 7fc5a6cfe700 5 -- op tracker -- seq: 46923, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) -7302> 2014-11-08 18:14:45.693118 7fc5b0316700 5 -- op tracker -- seq: 46923, time: 2014-11-08 18:14:45.693118, event: reached_pg, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) -7301> 2014-11-08 18:14:45.698616 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDPGPush(4.15s0 5773 [PushOp(4e7de15/1000002b702.00000065/head//4, version: 3473'6825, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4e7de15/1000002b702.00000065/head//4@3473'6825, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9000 con 0x1c5c2680 -7300> 2014-11-08 18:14:45.698666 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPush(4.15s2 5773 [PushOp(4e7de15/1000002b702.00000065/head//4, version: 3473'6825, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4e7de15/1000002b702.00000065/head//4@3473'6825, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3a8000 con 0x21468c60 -7299> 2014-11-08 18:14:45.698683 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- MOSDPGPush(4.15s1 5773 [PushOp(4e7de15/1000002b702.00000065/head//4, version: 3473'6825, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4e7de15/1000002b702.00000065/head//4@3473'6825, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3a9400 con 0x1c7911e0 -7298> 2014-11-08 18:14:45.705621 7fc5b0316700 5 -- op tracker -- seq: 46923, time: 2014-11-08 18:14:45.705621, event: done, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4406, attrs_read=1)) -7297> 2014-11-08 18:14:45.756079 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12050 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x1db192c0 con 0x1c5c2680 -7296> 2014-11-08 18:14:45.756126 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.755987, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -7295> 2014-11-08 18:14:45.756135 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.755991, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -7294> 2014-11-08 18:14:45.756137 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.756057, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -7293> 2014-11-08 18:14:45.756140 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.756120, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -7292> 2014-11-08 18:14:45.756143 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.756143, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -7291> 2014-11-08 18:14:45.756151 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.756151, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -7290> 2014-11-08 18:14:45.756157 7fc5b9b29700 5 -- op tracker -- seq: 46924, time: 2014-11-08 18:14:45.756157, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -7289> 2014-11-08 18:14:45.758527 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12051 ==== MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) v2 ==== 110+0+0 (2984444053 0 0) 0x229b9000 con 0x1c5c2680 -7288> 2014-11-08 18:14:45.758541 7fc5a6cfe700 5 -- op tracker -- seq: 46925, time: 2014-11-08 18:14:45.758470, event: header_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7287> 2014-11-08 18:14:45.758545 7fc5a6cfe700 5 -- op tracker -- seq: 46925, time: 2014-11-08 18:14:45.758472, event: throttled, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7286> 2014-11-08 18:14:45.758548 7fc5a6cfe700 5 -- op tracker -- seq: 46925, time: 2014-11-08 18:14:45.758523, event: all_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7285> 2014-11-08 18:14:45.758554 7fc5a6cfe700 5 -- op tracker -- seq: 46925, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7284> 2014-11-08 18:14:45.758591 7fc5adb11700 5 -- op tracker -- seq: 46925, time: 2014-11-08 18:14:45.758591, event: reached_pg, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7283> 2014-11-08 18:14:45.758626 7fc5adb11700 5 -- op tracker -- seq: 46925, time: 2014-11-08 18:14:45.758626, event: done, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7282> 2014-11-08 18:14:45.792337 7fc5a61fa700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 327 ==== MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) v2 ==== 110+0+0 (3470538995 0 0) 0x1d3a9400 con 0x1c7911e0 -7281> 2014-11-08 18:14:45.792384 7fc5a61fa700 5 -- op tracker -- seq: 46926, time: 2014-11-08 18:14:45.792313, event: header_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7280> 2014-11-08 18:14:45.792393 7fc5a61fa700 5 -- op tracker -- seq: 46926, time: 2014-11-08 18:14:45.792313, event: throttled, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7279> 2014-11-08 18:14:45.792396 7fc5a61fa700 5 -- op tracker -- seq: 46926, time: 2014-11-08 18:14:45.792336, event: all_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7278> 2014-11-08 18:14:45.792399 7fc5a61fa700 5 -- op tracker -- seq: 46926, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7277> 2014-11-08 18:14:45.792445 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 326 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x1ec20000 con 0x1c7911e0 -7276> 2014-11-08 18:14:45.792457 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792251, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -7275> 2014-11-08 18:14:45.792460 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792254, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -7274> 2014-11-08 18:14:45.792463 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792298, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -7273> 2014-11-08 18:14:45.792465 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792453, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -7272> 2014-11-08 18:14:45.792468 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792468, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -7271> 2014-11-08 18:14:45.792474 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792473, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -7270> 2014-11-08 18:14:45.792478 7fc5b9b29700 5 -- op tracker -- seq: 46927, time: 2014-11-08 18:14:45.792478, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -7269> 2014-11-08 18:14:45.792508 7fc5b0316700 5 -- op tracker -- seq: 46926, time: 2014-11-08 18:14:45.792508, event: reached_pg, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7268> 2014-11-08 18:14:45.792545 7fc5b0316700 5 -- op tracker -- seq: 46926, time: 2014-11-08 18:14:45.792545, event: done, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7267> 2014-11-08 18:14:45.857760 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.57s0 5773 ECSubReadReply(tid=25185, attrs_read=0)) v1 -- ?+0 0x21838780 con 0x1c5c2680 -7266> 2014-11-08 18:14:45.857810 7fc5ad310700 5 -- op tracker -- seq: 46901, time: 2014-11-08 18:14:45.857809, event: done, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25185, to_read={69060857/1000002bb90.00000021/head//4=0,4194304,4070857/1000002ac85.00000004/head//4=0,4194304,e5170857/1000002992d.0000000a/head//4=0,4194304}, attrs_to_read=)) -7265> 2014-11-08 18:14:45.902220 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3673 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:45.899516) v2 ==== 47+0+0 (1971623803 0 0) 0x224a3fc0 con 0x1cce2ec0 -7264> 2014-11-08 18:14:45.902249 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.899516) v2 -- ?+0 0x224a23a0 con 0x1cce2ec0 -7263> 2014-11-08 18:14:45.902593 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3673 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:45.899516) v2 ==== 47+0+0 (1971623803 0 0) 0x23147620 con 0x1cce3440 -7262> 2014-11-08 18:14:45.902605 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:45.899516) v2 -- ?+0 0x23146900 con 0x1cce3440 -7261> 2014-11-08 18:14:45.912939 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1592 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x1eae1e00 con 0x21468c60 -7260> 2014-11-08 18:14:45.912967 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.912840, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -7259> 2014-11-08 18:14:45.912975 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.912843, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -7258> 2014-11-08 18:14:45.912978 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.912921, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -7257> 2014-11-08 18:14:45.912982 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.912961, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -7256> 2014-11-08 18:14:45.912987 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.912987, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -7255> 2014-11-08 18:14:45.912997 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.912997, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -7254> 2014-11-08 18:14:45.913004 7fc5b9b29700 5 -- op tracker -- seq: 46928, time: 2014-11-08 18:14:45.913004, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -7253> 2014-11-08 18:14:46.025025 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpReadReply(4.b2s1 5773 ECSubReadReply(tid=20429, attrs_read=0)) v1 -- ?+0 0x21a4f580 con 0x1c5c0580 -7252> 2014-11-08 18:14:46.025704 7fc5acb0f700 5 -- op tracker -- seq: 46904, time: 2014-11-08 18:14:46.025704, event: done, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20429, to_read={90285b2/10000029d18.00000007/head//4=0,4194304,a65285b2/100000199ba.00000071/head//4=0,4194304,d80385b2/1000002b051.00000027/head//4=0,4194304,197685b2/10000029638.00000050/head//4=0,4194304,d3785b2/1000002b48d.00000054/head//4=0,4194304}, attrs_to_read=)) -7251> 2014-11-08 18:14:46.025755 7fc5acb0f700 5 -- op tracker -- seq: 46922, time: 2014-11-08 18:14:46.025755, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7250> 2014-11-08 18:14:46.025771 7fc5acb0f700 5 -- op tracker -- seq: 46922, time: 2014-11-08 18:14:46.025771, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7249> 2014-11-08 18:14:46.025827 7fc5acb0f700 5 -- op tracker -- seq: 46922, time: 2014-11-08 18:14:46.025827, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb dfcb90b7/1000001b501.00000ad0/head//4) -7248> 2014-11-08 18:14:46.025847 7fc5acb0f700 5 -- op tracker -- seq: 46915, time: 2014-11-08 18:14:46.025847, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) -7247> 2014-11-08 18:14:46.079705 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59245, attrs_read=0)) v1 -- ?+0 0x21a4b200 con 0x21468c60 -7246> 2014-11-08 18:14:46.079761 7fc5acb0f700 5 -- op tracker -- seq: 46915, time: 2014-11-08 18:14:46.079760, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59245, to_read={3b0414e/1000001ac5e.0000002a/head//4=0,4194304}, attrs_to_read=)) -7245> 2014-11-08 18:14:46.394100 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2213 ==== MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 4196083+0+0 (2742014035 0 0) 0x21613c00 con 0x1e50aec0 -7244> 2014-11-08 18:14:46.394211 7fc5a56ef700 5 -- op tracker -- seq: 46929, time: 2014-11-08 18:14:46.176877, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7243> 2014-11-08 18:14:46.394232 7fc5a56ef700 5 -- op tracker -- seq: 46929, time: 2014-11-08 18:14:46.176882, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7242> 2014-11-08 18:14:46.394249 7fc5a56ef700 5 -- op tracker -- seq: 46929, time: 2014-11-08 18:14:46.394090, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7241> 2014-11-08 18:14:46.394264 7fc5a56ef700 5 -- op tracker -- seq: 46929, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7240> 2014-11-08 18:14:46.394318 7fc5acb0f700 5 -- op tracker -- seq: 46929, time: 2014-11-08 18:14:46.394318, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7239> 2014-11-08 18:14:46.394429 7fc5acb0f700 5 -- op tracker -- seq: 46929, time: 2014-11-08 18:14:46.394429, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(4cdb90b7/100000297c2.0000007c/head//4, version: 3033'4375, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdb90b7/100000297c2.0000007c/head//4@3033'4375, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42eb90b7/1000002a8e5.0000005c/head//4, version: 3458'5521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42eb90b7/1000002a8e5.0000005c/head//4@3458'5521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7238> 2014-11-08 18:14:46.484145 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x22454560 con 0x1e50aec0 -7237> 2014-11-08 18:14:46.484181 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x22456540 con 0x1e50aec0 -7236> 2014-11-08 18:14:46.484191 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(4cdb90b7/100000297c2.0000007c/head//4),PushReplyOp(42eb90b7/1000002a8e5.0000005c/head//4)]) v2 -- ?+0 0x24840c00 con 0x1e50aec0 -7235> 2014-11-08 18:14:46.514836 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3735 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:46.514350) v2 ==== 47+0+0 (635337611 0 0) 0x236841a0 con 0x1cce3180 -7234> 2014-11-08 18:14:46.514865 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:46.514350) v2 -- ?+0 0x224a3fc0 con 0x1cce3180 -7233> 2014-11-08 18:14:46.514892 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3735 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:46.514350) v2 ==== 47+0+0 (635337611 0 0) 0x2286a1c0 con 0x1cce3700 -7232> 2014-11-08 18:14:46.514902 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:46.514350) v2 -- ?+0 0x23147620 con 0x1cce3700 -7231> 2014-11-08 18:14:46.593048 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.8fs0 5773 ECSubReadReply(tid=59244, attrs_read=0)) v1 -- ?+0 0x1c899b80 con 0x21468c60 -7230> 2014-11-08 18:14:46.593105 7fc5af314700 5 -- op tracker -- seq: 46910, time: 2014-11-08 18:14:46.593104, event: done, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59244, to_read={67fa678f/1000001a0ef.00000236/head//4=0,4194304,c21b678f/1000001aa8d.0000011c/head//4=0,4194304,87ab678f/1000002a2df.00000064/head//4=0,4194304,f06c678f/1000002bca6.0000013f/head//4=0,4194304,846d678f/1000002afc4.00000068/head//4=0,4194304}, attrs_to_read=)) -7229> 2014-11-08 18:14:46.670194 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1593 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (95963031 0 0) 0x1ce60b40 con 0x21468c60 -7228> 2014-11-08 18:14:46.670226 7fc5a39d2700 5 -- op tracker -- seq: 46930, time: 2014-11-08 18:14:46.670105, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) -7227> 2014-11-08 18:14:46.670238 7fc5a39d2700 5 -- op tracker -- seq: 46930, time: 2014-11-08 18:14:46.670108, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) -7226> 2014-11-08 18:14:46.670244 7fc5a39d2700 5 -- op tracker -- seq: 46930, time: 2014-11-08 18:14:46.670190, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) -7225> 2014-11-08 18:14:46.670249 7fc5a39d2700 5 -- op tracker -- seq: 46930, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) -7224> 2014-11-08 18:14:46.670288 7fc5acb0f700 5 -- op tracker -- seq: 46930, time: 2014-11-08 18:14:46.670287, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) -7223> 2014-11-08 18:14:46.672815 7fc5c362f700 5 osd.9 5773 tick -7222> 2014-11-08 18:14:46.729032 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59247, attrs_read=0)) v1 -- ?+0 0x21a4ee00 con 0x21468c60 -7221> 2014-11-08 18:14:46.729095 7fc5acb0f700 5 -- op tracker -- seq: 46930, time: 2014-11-08 18:14:46.729094, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59247, to_read={27a1414e/1000001b39d.00000040/head//4=0,4194304}, attrs_to_read=)) -7220> 2014-11-08 18:14:46.821934 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8514 ==== MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294110+0+0 (2143204170 0 0) 0x24847200 con 0x1c5c0580 -7219> 2014-11-08 18:14:46.821983 7fc5a152e700 5 -- op tracker -- seq: 46931, time: 2014-11-08 18:14:46.661070, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7218> 2014-11-08 18:14:46.822029 7fc5a152e700 5 -- op tracker -- seq: 46931, time: 2014-11-08 18:14:46.661075, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7217> 2014-11-08 18:14:46.822044 7fc5a152e700 5 -- op tracker -- seq: 46931, time: 2014-11-08 18:14:46.821924, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7216> 2014-11-08 18:14:46.822058 7fc5a152e700 5 -- op tracker -- seq: 46931, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7215> 2014-11-08 18:14:46.822153 7fc5adb11700 5 -- op tracker -- seq: 46931, time: 2014-11-08 18:14:46.822153, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7214> 2014-11-08 18:14:46.822286 7fc5adb11700 5 -- op tracker -- seq: 46931, time: 2014-11-08 18:14:46.822286, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(7609fcb5/1000002a652.00000006/head//4, version: 3159'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7609fcb5/1000002a652.00000006/head//4@3159'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cc79fcb5/1000002b92f.00000056/head//4, version: 3473'7061, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cc79fcb5/1000002b92f.00000056/head//4@3473'7061, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5c9fcb5/1000002bd56.0000007a/head//4, version: 4864'8143, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5c9fcb5/1000002bd56.0000007a/head//4@4864'8143, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7213> 2014-11-08 18:14:47.043626 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -7212> 2014-11-08 18:14:47.043648 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1e5379e0 con 0x1cae2940 -7211> 2014-11-08 18:14:47.043666 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x25fc5be0 con 0x1cae2680 -7210> 2014-11-08 18:14:47.043678 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x23142d00 con 0x1c5c02c0 -7209> 2014-11-08 18:14:47.043697 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x239e90e0 con 0x1cae0160 -7208> 2014-11-08 18:14:47.043707 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x25fc14a0 con 0x1cae0f20 -7207> 2014-11-08 18:14:47.043716 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x239eddc0 con 0x1cae0c60 -7206> 2014-11-08 18:14:47.043726 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x227e4920 con 0x1cae23c0 -7205> 2014-11-08 18:14:47.043736 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1db1f620 con 0x1cae56a0 -7204> 2014-11-08 18:14:47.043796 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1db1eea0 con 0x1cbf9ce0 -7203> 2014-11-08 18:14:47.043808 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x213b8f00 con 0x1d603020 -7202> 2014-11-08 18:14:47.043817 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1ec21a40 con 0x1dbd9b80 -7201> 2014-11-08 18:14:47.043827 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x227ab660 con 0x1d235d80 -7200> 2014-11-08 18:14:47.043837 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1dbfb660 con 0x1cae1fa0 -7199> 2014-11-08 18:14:47.043846 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1dbf90e0 con 0x1cae1ce0 -7198> 2014-11-08 18:14:47.043867 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1dbfeae0 con 0x1cae1340 -7197> 2014-11-08 18:14:47.043877 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1dbf8960 con 0x1cae1080 -7196> 2014-11-08 18:14:47.043886 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x233f9860 con 0x1cae1a20 -7195> 2014-11-08 18:14:47.043896 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:47.043645) v2 -- ?+0 0x1dbfa760 con 0x1cae2c00 -7194> 2014-11-08 18:14:47.044481 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x1f4c3a20 con 0x1cae2680 -7193> 2014-11-08 18:14:47.044654 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x2583f800 con 0x1cae56a0 -7192> 2014-11-08 18:14:47.044696 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x257c0d20 con 0x1c5c02c0 -7191> 2014-11-08 18:14:47.044711 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x2290d820 con 0x1cae0160 -7190> 2014-11-08 18:14:47.044727 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x227e6cc0 con 0x1cae0f20 -7189> 2014-11-08 18:14:47.044738 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x1e890f00 con 0x1cae0c60 -7188> 2014-11-08 18:14:47.044746 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x25e86540 con 0x1cae1340 -7187> 2014-11-08 18:14:47.044755 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x1decc740 con 0x1cae1080 -7186> 2014-11-08 18:14:47.044763 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x1eaca940 con 0x1cae1fa0 -7185> 2014-11-08 18:14:47.044772 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x4533480 con 0x1cae1ce0 -7184> 2014-11-08 18:14:47.044780 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x1d325fa0 con 0x1cae1a20 -7183> 2014-11-08 18:14:47.044789 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x236879e0 con 0x1cae2940 -7182> 2014-11-08 18:14:47.044798 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x2495e720 con 0x1cae23c0 -7181> 2014-11-08 18:14:47.044882 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 307 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x227ce900 con 0x1d235d80 -7180> 2014-11-08 18:14:47.044896 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 307 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x1db1cb00 con 0x1dbd9b80 -7179> 2014-11-08 18:14:47.045248 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3720 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x2127f800 con 0x1cae2c00 -7178> 2014-11-08 18:14:47.053026 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 500 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x246cf080 con 0x1cbf9ce0 -7177> 2014-11-08 18:14:47.053183 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 500 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.043645) v2 ==== 47+0+0 (2794350707 0 0) 0x2234ab20 con 0x1d603020 -7176> 2014-11-08 18:14:47.083212 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8515 ==== MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b7336b16/ 100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (2674560309 0 0) 0x1ebb5c00 con 0x1c5c0580 -7175> 2014-11-08 18:14:47.083269 7fc5a152e700 5 -- op tracker -- seq: 46932, time: 2014-11-08 18:14:46.865174, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( b7336b16/100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7174> 2014-11-08 18:14:47.083295 7fc5a152e700 5 -- op tracker -- seq: 46932, time: 2014-11-08 18:14:46.865178, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b7336b16/ 100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7173> 2014-11-08 18:14:47.083316 7fc5a152e700 5 -- op tracker -- seq: 46932, time: 2014-11-08 18:14:47.083201, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b7336b16/ 100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7172> 2014-11-08 18:14:47.083335 7fc5a152e700 5 -- op tracker -- seq: 46932, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b7336b16/ 100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7171> 2014-11-08 18:14:47.083483 7fc5afb15700 5 -- op tracker -- seq: 46932, time: 2014-11-08 18:14:47.083483, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b7336b16/ 100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7170> 2014-11-08 18:14:47.083776 7fc5afb15700 5 -- op tracker -- seq: 46932, time: 2014-11-08 18:14:47.083775, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(aae16b16/100000197e6.00000050/head//4, version: 397'765, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(aae16b16/100000197e6.00000050/head//4@397'765, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8e026b16/10000029631.00000002/head//4, version: 3033'4220, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8e026b16/10000029631.00000002/head//4@3033'4220, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(eff26b16/1000001b2c3.00000051/head//4, version: 2829'2729, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(eff26b16/1000001b2c3.00000051/head//4@2829'2729, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b7336b16/100000283f5.0000008e/head//4, version: 3033'3693, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b7336b16/ 100000283f5.0000008e/head//4@3033'3693, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(55436b16/10000019fe3.00000043/head//4, version: 397'1515, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(55436b16/10000019fe3.00000043/head//4@397'1515, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7169> 2014-11-08 18:14:47.129103 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8516 ==== MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (3100372565 0 0) 0x224ac140 con 0x1c5c0580 -7168> 2014-11-08 18:14:47.129140 7fc5a152e700 5 -- op tracker -- seq: 46933, time: 2014-11-08 18:14:47.128967, event: header_read, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) -7167> 2014-11-08 18:14:47.129153 7fc5a152e700 5 -- op tracker -- seq: 46933, time: 2014-11-08 18:14:47.128971, event: throttled, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) -7166> 2014-11-08 18:14:47.129161 7fc5a152e700 5 -- op tracker -- seq: 46933, time: 2014-11-08 18:14:47.129094, event: all_read, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) -7165> 2014-11-08 18:14:47.129167 7fc5a152e700 5 -- op tracker -- seq: 46933, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) -7164> 2014-11-08 18:14:47.129224 7fc5af314700 5 -- op tracker -- seq: 46933, time: 2014-11-08 18:14:47.129224, event: reached_pg, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) -7163> 2014-11-08 18:14:47.129229 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8517 ==== MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) v1 ==== 219+0+0 (3837841136 0 0) 0x224ad100 con 0x1c5c0580 -7162> 2014-11-08 18:14:47.129247 7fc5a152e700 5 -- op tracker -- seq: 46934, time: 2014-11-08 18:14:47.129201, event: header_read, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) -7161> 2014-11-08 18:14:47.129251 7fc5a152e700 5 -- op tracker -- seq: 46934, time: 2014-11-08 18:14:47.129201, event: throttled, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) -7160> 2014-11-08 18:14:47.129256 7fc5a152e700 5 -- op tracker -- seq: 46934, time: 2014-11-08 18:14:47.129226, event: all_read, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) -7159> 2014-11-08 18:14:47.129259 7fc5a152e700 5 -- op tracker -- seq: 46934, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) -7158> 2014-11-08 18:14:47.129282 7fc5acb0f700 5 -- op tracker -- seq: 46934, time: 2014-11-08 18:14:47.129282, event: reached_pg, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) -7157> 2014-11-08 18:14:47.236933 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22453fc0 con 0x1c5c0580 -7156> 2014-11-08 18:14:47.237134 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22456ae0 con 0x1c5c0580 -7155> 2014-11-08 18:14:47.237169 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x22450b40 con 0x1c5c0580 -7154> 2014-11-08 18:14:47.237198 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(7609fcb5/1000002a652.00000006/head//4),PushReplyOp(cc79fcb5/1000002b92f.00000056/head//4),PushReplyOp(e5c9fcb5/1000002bd56.0000007a/head//4)]) v2 -- ?+0 0x229b9000 con 0x1c5c0580 -7153> 2014-11-08 18:14:47.240554 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8518 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) v3 ==== 757+0+0 (2478063234 0 0) 0x1ce9ba80 con 0x1c5c0580 -7152> 2014-11-08 18:14:47.240591 7fc5a152e700 5 -- op tracker -- seq: 46935, time: 2014-11-08 18:14:47.240457, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7151> 2014-11-08 18:14:47.240601 7fc5a152e700 5 -- op tracker -- seq: 46935, time: 2014-11-08 18:14:47.240460, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7150> 2014-11-08 18:14:47.240607 7fc5a152e700 5 -- op tracker -- seq: 46935, time: 2014-11-08 18:14:47.240545, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7149> 2014-11-08 18:14:47.240612 7fc5a152e700 5 -- op tracker -- seq: 46935, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7148> 2014-11-08 18:14:47.240681 7fc5b0316700 5 -- op tracker -- seq: 46935, time: 2014-11-08 18:14:47.240681, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7147> 2014-11-08 18:14:47.240700 7fc5b0316700 5 -- op tracker -- seq: 46935, time: 2014-11-08 18:14:47.240700, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7146> 2014-11-08 18:14:47.240745 7fc5b0316700 5 -- op tracker -- seq: 46935, time: 2014-11-08 18:14:47.240745, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb e5c9fcb5/1000002bd56.0000007a/head//4) -7145> 2014-11-08 18:14:47.284629 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1594 ==== MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f06c678f/ 1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (2071135812 0 0) 0x1d3a8000 con 0x21468c60 -7144> 2014-11-08 18:14:47.285123 7fc5a39d2700 5 -- op tracker -- seq: 46936, time: 2014-11-08 18:14:46.996184, event: header_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f06c678f/1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7143> 2014-11-08 18:14:47.285151 7fc5a39d2700 5 -- op tracker -- seq: 46936, time: 2014-11-08 18:14:46.996187, event: throttled, op: MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( f06c678f/1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7142> 2014-11-08 18:14:47.285172 7fc5a39d2700 5 -- op tracker -- seq: 46936, time: 2014-11-08 18:14:47.284617, event: all_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f06c678f/ 1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7141> 2014-11-08 18:14:47.285190 7fc5a39d2700 5 -- op tracker -- seq: 46936, time: 0.000000, event: dispatched, op: MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f06c678f/ 1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7140> 2014-11-08 18:14:47.285266 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1595 ==== MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) v2 ==== 110+0+0 (2891160102 0 0) 0x1d3af400 con 0x21468c60 -7139> 2014-11-08 18:14:47.285275 7fc5a39d2700 5 -- op tracker -- seq: 46937, time: 2014-11-08 18:14:47.285236, event: header_read, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) -7138> 2014-11-08 18:14:47.285278 7fc5a39d2700 5 -- op tracker -- seq: 46937, time: 2014-11-08 18:14:47.285237, event: throttled, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) -7137> 2014-11-08 18:14:47.285282 7fc5a39d2700 5 -- op tracker -- seq: 46937, time: 2014-11-08 18:14:47.285263, event: all_read, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) -7136> 2014-11-08 18:14:47.285285 7fc5a39d2700 5 -- op tracker -- seq: 46937, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) -7135> 2014-11-08 18:14:47.285326 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1596 ==== MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) v2 ==== 110+0+0 (2290917758 0 0) 0x21610000 con 0x21468c60 -7134> 2014-11-08 18:14:47.285334 7fc5a39d2700 5 -- op tracker -- seq: 46938, time: 2014-11-08 18:14:47.285301, event: header_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7133> 2014-11-08 18:14:47.285338 7fc5a39d2700 5 -- op tracker -- seq: 46938, time: 2014-11-08 18:14:47.285302, event: throttled, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7132> 2014-11-08 18:14:47.285340 7fc5a39d2700 5 -- op tracker -- seq: 46938, time: 2014-11-08 18:14:47.285326, event: all_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7131> 2014-11-08 18:14:47.285344 7fc5a39d2700 5 -- op tracker -- seq: 46938, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7130> 2014-11-08 18:14:47.286500 7fc5ad310700 5 -- op tracker -- seq: 46937, time: 2014-11-08 18:14:47.286500, event: reached_pg, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) -7129> 2014-11-08 18:14:47.286578 7fc5ad310700 5 -- op tracker -- seq: 46937, time: 2014-11-08 18:14:47.286578, event: done, op: MOSDPGPushReply(4.cs0 5773 [PushReplyOp(fe393e0c/1000001b50a.000000f6/head//4)]) -7128> 2014-11-08 18:14:47.286614 7fc5adb11700 5 -- op tracker -- seq: 46938, time: 2014-11-08 18:14:47.286614, event: reached_pg, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7127> 2014-11-08 18:14:47.286651 7fc5adb11700 5 -- op tracker -- seq: 46938, time: 2014-11-08 18:14:47.286651, event: done, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(4e7de15/1000002b702.00000065/head//4)]) -7126> 2014-11-08 18:14:47.296552 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 493 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.293656) v2 ==== 47+0+0 (659447794 0 0) 0x24880960 con 0x1eb9c8e0 -7125> 2014-11-08 18:14:47.296601 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.293656) v2 -- ?+0 0x2286a1c0 con 0x1eb9c8e0 -7124> 2014-11-08 18:14:47.296552 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 493 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.293656) v2 ==== 47+0+0 (659447794 0 0) 0x22e5e720 con 0x1eb9d800 -7123> 2014-11-08 18:14:47.296628 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.293656) v2 -- ?+0 0x236841a0 con 0x1eb9d800 -7122> 2014-11-08 18:14:47.307248 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1502 ==== pg_stats_ack(2 pgs tid 1327) v1 ==== 62+0+0 (2795600044 0 0) 0x2286a3a0 con 0x44ed6a0 -7121> 2014-11-08 18:14:47.311680 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1597 ==== pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) v3 ==== 757+0+0 (2690661724 0 0) 0x2366f980 con 0x21468c60 -7120> 2014-11-08 18:14:47.311738 7fc5a39d2700 5 -- op tracker -- seq: 46939, time: 2014-11-08 18:14:47.311571, event: header_read, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7119> 2014-11-08 18:14:47.311752 7fc5a39d2700 5 -- op tracker -- seq: 46939, time: 2014-11-08 18:14:47.311575, event: throttled, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7118> 2014-11-08 18:14:47.311762 7fc5a39d2700 5 -- op tracker -- seq: 46939, time: 2014-11-08 18:14:47.311660, event: all_read, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7117> 2014-11-08 18:14:47.311771 7fc5a39d2700 5 -- op tracker -- seq: 46939, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7116> 2014-11-08 18:14:47.311815 7fc5b0b17700 5 -- op tracker -- seq: 46939, time: 2014-11-08 18:14:47.311815, event: reached_pg, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7115> 2014-11-08 18:14:47.311825 7fc5b0b17700 5 -- op tracker -- seq: 46939, time: 2014-11-08 18:14:47.311825, event: started, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7114> 2014-11-08 18:14:47.312028 7fc5a39d2700 5 -- op tracker -- seq: 46939, time: 2014-11-08 18:14:47.312028, event: done, op: pg_backfill(progress 4.91s1 e 5773/5773 lb eaccf291/1000001b4fd.00000128/head//4) -7113> 2014-11-08 18:14:47.331489 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpReadReply(4.bcs0 5773 ECSubReadReply(tid=20435, attrs_read=0)) v1 -- ?+0 0x21a48500 con 0x1c5c0580 -7112> 2014-11-08 18:14:47.331524 7fc5acb0f700 5 -- op tracker -- seq: 46934, time: 2014-11-08 18:14:47.331523, event: done, op: MOSDECSubOpRead(4.bcs1 5773 ECSubRead(tid=20435, to_read={7c4617bc/1000001b4f3.00000078/head//4=0,4194304,da1717bc/1000002992e.00000016/head//4=0,4194304}, attrs_to_read=)) -7111> 2014-11-08 18:14:47.331594 7fc5acb0f700 5 -- op tracker -- seq: 46936, time: 2014-11-08 18:14:47.331594, event: reached_pg, op: MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f06c678f/1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7110> 2014-11-08 18:14:47.332024 7fc5acb0f700 5 -- op tracker -- seq: 46936, time: 2014-11-08 18:14:47.332023, event: done, op: MOSDPGPush(4.8fs2 5773 [PushOp(67fa678f/1000001a0ef.00000236/head//4, version: 397'1633, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67fa678f/1000001a0ef.00000236/head//4@397'1633, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c21b678f/1000001aa8d.0000011c/head//4, version: 2829'2342, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c21b678f/1000001aa8d.0000011c/head//4@2829'2342, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87ab678f/1000002a2df.00000064/head//4, version: 3033'4616, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87ab678f/1000002a2df.00000064/head//4@3033'4616, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f06c678f/1000002bca6.0000013f/head//4, version: 3485'7406, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f06c678f/ 1000002bca6.0000013f/head//4@3485'7406, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(846d678f/1000002afc4.00000068/head//4, version: 4157'7581, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(846d678f/1000002afc4.00000068/head//4@4157'7581, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7109> 2014-11-08 18:14:47.361760 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12052 ==== pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) v3 ==== 757+0+0 (2640670562 0 0) 0x1e58a400 con 0x1c5c2680 -7108> 2014-11-08 18:14:47.361781 7fc5a6cfe700 5 -- op tracker -- seq: 46940, time: 2014-11-08 18:14:47.361660, event: header_read, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7107> 2014-11-08 18:14:47.361806 7fc5a6cfe700 5 -- op tracker -- seq: 46940, time: 2014-11-08 18:14:47.361664, event: throttled, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7106> 2014-11-08 18:14:47.361810 7fc5a6cfe700 5 -- op tracker -- seq: 46940, time: 2014-11-08 18:14:47.361753, event: all_read, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7105> 2014-11-08 18:14:47.361813 7fc5a6cfe700 5 -- op tracker -- seq: 46940, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7104> 2014-11-08 18:14:47.361871 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12053 ==== MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3481588389 0 0) 0x1bea7500 con 0x1c5c2680 -7103> 2014-11-08 18:14:47.361880 7fc5a6cfe700 5 -- op tracker -- seq: 46941, time: 2014-11-08 18:14:47.361843, event: header_read, op: MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) -7102> 2014-11-08 18:14:47.361884 7fc5a6cfe700 5 -- op tracker -- seq: 46941, time: 2014-11-08 18:14:47.361843, event: throttled, op: MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) -7101> 2014-11-08 18:14:47.361887 7fc5a6cfe700 5 -- op tracker -- seq: 46941, time: 2014-11-08 18:14:47.361870, event: all_read, op: MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) -7100> 2014-11-08 18:14:47.361890 7fc5a6cfe700 5 -- op tracker -- seq: 46941, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) -7099> 2014-11-08 18:14:47.361927 7fc5acb0f700 5 -- op tracker -- seq: 46940, time: 2014-11-08 18:14:47.361927, event: reached_pg, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7098> 2014-11-08 18:14:47.361936 7fc5acb0f700 5 -- op tracker -- seq: 46940, time: 2014-11-08 18:14:47.361936, event: started, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7097> 2014-11-08 18:14:47.361983 7fc5acb0f700 5 -- op tracker -- seq: 46940, time: 2014-11-08 18:14:47.361983, event: done, op: pg_backfill(progress 4.67s1 e 5773/5773 lb c90ff67/1000001b40e.00000016/head//4) -7096> 2014-11-08 18:14:47.361997 7fc5acb0f700 5 -- op tracker -- seq: 46941, time: 2014-11-08 18:14:47.361997, event: reached_pg, op: MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) -7095> 2014-11-08 18:14:47.414613 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22457440 con 0x1c5c0580 -7094> 2014-11-08 18:14:47.414714 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22456ea0 con 0x1c5c0580 -7093> 2014-11-08 18:14:47.414722 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22453c00 con 0x1c5c0580 -7092> 2014-11-08 18:14:47.414728 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22451a40 con 0x1c5c0580 -7091> 2014-11-08 18:14:47.414733 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22455a00 con 0x1c5c0580 -7090> 2014-11-08 18:14:47.414742 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(aae16b16/100000197e6.00000050/head//4),PushReplyOp(8e026b16/10000029631.00000002/head//4),PushReplyOp(eff26b16/1000001b2c3.00000051/head//4),PushReplyOp(b7336b16/100000283f5.0000008e/head//4),PushReplyOp(55436b16/10000019fe3.00000043/head//4)]) v2 -- ?+0 0x21612c00 con 0x1c5c0580 -7089> 2014-11-08 18:14:47.446850 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8519 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) v3 ==== 757+0+0 (4157691220 0 0) 0x1e58de80 con 0x1c5c0580 -7088> 2014-11-08 18:14:47.446884 7fc5a152e700 5 -- op tracker -- seq: 46942, time: 2014-11-08 18:14:47.446757, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7087> 2014-11-08 18:14:47.446892 7fc5a152e700 5 -- op tracker -- seq: 46942, time: 2014-11-08 18:14:47.446761, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7086> 2014-11-08 18:14:47.446896 7fc5a152e700 5 -- op tracker -- seq: 46942, time: 2014-11-08 18:14:47.446841, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7085> 2014-11-08 18:14:47.446899 7fc5a152e700 5 -- op tracker -- seq: 46942, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7084> 2014-11-08 18:14:47.446943 7fc5afb15700 5 -- op tracker -- seq: 46942, time: 2014-11-08 18:14:47.446942, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7083> 2014-11-08 18:14:47.446953 7fc5afb15700 5 -- op tracker -- seq: 46942, time: 2014-11-08 18:14:47.446953, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7082> 2014-11-08 18:14:47.447031 7fc5afb15700 5 -- op tracker -- seq: 46942, time: 2014-11-08 18:14:47.447031, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 55436b16/10000019fe3.00000043/head//4) -7081> 2014-11-08 18:14:47.447108 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpRead(4.15s2 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=c958de15/10000028657.00000051/head//4,2968de15/1000002a90f.00000070/head//4)) v1 -- ?+0 0x230298c0 con 0x1c5c2680 -7080> 2014-11-08 18:14:47.447127 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2302e780 con 0x42c86e0 -7079> 2014-11-08 18:14:47.447141 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=2 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 31c8de15/1000002b072.0000009c/head//4 -7078> 2014-11-08 18:14:47.447164 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_backfill(progress 4.15s0 e 5773/5773 lb 4e7de15/1000002b702.00000065/head//4) v3 -- ?+0 0x4275100 con 0x1c5c2680 -7077> 2014-11-08 18:14:47.447172 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_backfill(progress 4.15s2 e 5773/5773 lb 4e7de15/1000002b702.00000065/head//4) v3 -- ?+0 0x226f4380 con 0x21468c60 -7076> 2014-11-08 18:14:47.447183 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_backfill(progress 4.15s1 e 5773/5773 lb 4e7de15/1000002b702.00000065/head//4) v3 -- ?+0 0x1e58e780 con 0x1c7911e0 -7075> 2014-11-08 18:14:47.447206 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2302e780 con 0x42c86e0 -7074> 2014-11-08 18:14:47.447218 7fc5b9328700 5 -- op tracker -- seq: 46943, time: 2014-11-08 18:14:47.447134, event: header_read, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) -7073> 2014-11-08 18:14:47.447226 7fc5b9328700 5 -- op tracker -- seq: 46943, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) -7072> 2014-11-08 18:14:47.447230 7fc5b9328700 5 -- op tracker -- seq: 46943, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) -7071> 2014-11-08 18:14:47.447233 7fc5b9328700 5 -- op tracker -- seq: 46943, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) -7070> 2014-11-08 18:14:47.447353 7fc5b0316700 5 -- op tracker -- seq: 46943, time: 2014-11-08 18:14:47.447353, event: reached_pg, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) -7069> 2014-11-08 18:14:47.625717 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1598 ==== MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3602359672 0 0) 0x23938c00 con 0x21468c60 -7068> 2014-11-08 18:14:47.625759 7fc5a39d2700 5 -- op tracker -- seq: 46944, time: 2014-11-08 18:14:47.561550, event: header_read, op: MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7067> 2014-11-08 18:14:47.625773 7fc5a39d2700 5 -- op tracker -- seq: 46944, time: 2014-11-08 18:14:47.561554, event: throttled, op: MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7066> 2014-11-08 18:14:47.625781 7fc5a39d2700 5 -- op tracker -- seq: 46944, time: 2014-11-08 18:14:47.625706, event: all_read, op: MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7065> 2014-11-08 18:14:47.625788 7fc5a39d2700 5 -- op tracker -- seq: 46944, time: 0.000000, event: dispatched, op: MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7064> 2014-11-08 18:14:47.625829 7fc5ae312700 5 -- op tracker -- seq: 46944, time: 2014-11-08 18:14:47.625829, event: reached_pg, op: MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7063> 2014-11-08 18:14:47.625914 7fc5ae312700 5 -- op tracker -- seq: 46944, time: 2014-11-08 18:14:47.625914, event: done, op: MOSDPGPush(4.91s1 5773 [PushOp(853ff291/100000296ee.0000004b/head//4, version: 3033'4191, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(853ff291/100000296ee.0000004b/head//4@3033'4191, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7062> 2014-11-08 18:14:47.651937 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1599 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (1392530742 0 0) 0x1ce62640 con 0x21468c60 -7061> 2014-11-08 18:14:47.651979 7fc5a39d2700 5 -- op tracker -- seq: 46945, time: 2014-11-08 18:14:47.651822, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) -7060> 2014-11-08 18:14:47.651992 7fc5a39d2700 5 -- op tracker -- seq: 46945, time: 2014-11-08 18:14:47.651826, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) -7059> 2014-11-08 18:14:47.651999 7fc5a39d2700 5 -- op tracker -- seq: 46945, time: 2014-11-08 18:14:47.651928, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) -7058> 2014-11-08 18:14:47.652005 7fc5a39d2700 5 -- op tracker -- seq: 46945, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) -7057> 2014-11-08 18:14:47.666513 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.67s1 5773 ECSubReadReply(tid=25187, attrs_read=0)) v1 -- ?+0 0x21a4bc00 con 0x1c5c2680 -7056> 2014-11-08 18:14:47.666560 7fc5acb0f700 5 -- op tracker -- seq: 46941, time: 2014-11-08 18:14:47.666560, event: done, op: MOSDECSubOpRead(4.67s2 5773 ECSubRead(tid=25187, to_read={19b1ff67/1000001a111.0000003f/head//4=0,4194304}, attrs_to_read=)) -7055> 2014-11-08 18:14:47.666595 7fc5acb0f700 5 -- op tracker -- seq: 46945, time: 2014-11-08 18:14:47.666595, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) -7054> 2014-11-08 18:14:47.672959 7fc5c362f700 5 osd.9 5773 tick -7053> 2014-11-08 18:14:47.692454 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) v1 -- ?+0 0x1c89bc00 con 0x42c86e0 -7052> 2014-11-08 18:14:47.692502 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1c89bc00 con 0x42c86e0 -7051> 2014-11-08 18:14:47.692504 7fc5b0316700 5 -- op tracker -- seq: 46943, time: 2014-11-08 18:14:47.692504, event: done, op: MOSDECSubOpRead(4.15s1 5773 ECSubRead(tid=4408, to_read={c958de15/10000028657.00000051/head//4=0,4194304,2968de15/1000002a90f.00000070/head//4=0,4194304}, attrs_to_read=)) -7050> 2014-11-08 18:14:47.692518 7fc5b9328700 5 -- op tracker -- seq: 46946, time: 2014-11-08 18:14:47.692491, event: header_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) -7049> 2014-11-08 18:14:47.692522 7fc5b9328700 5 -- op tracker -- seq: 46946, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) -7048> 2014-11-08 18:14:47.692524 7fc5b9328700 5 -- op tracker -- seq: 46946, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) -7047> 2014-11-08 18:14:47.692526 7fc5b9328700 5 -- op tracker -- seq: 46946, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) -7046> 2014-11-08 18:14:47.692553 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22455280 con 0x21468c60 -7045> 2014-11-08 18:14:47.692561 7fc5adb11700 5 -- op tracker -- seq: 46946, time: 2014-11-08 18:14:47.692561, event: reached_pg, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) -7044> 2014-11-08 18:14:47.692567 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22455be0 con 0x21468c60 -7043> 2014-11-08 18:14:47.692574 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x224521c0 con 0x21468c60 -7042> 2014-11-08 18:14:47.692579 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22452d00 con 0x21468c60 -7041> 2014-11-08 18:14:47.692585 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x22452940 con 0x21468c60 -7040> 2014-11-08 18:14:47.692591 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.8fs0 5773 [PushReplyOp(67fa678f/1000001a0ef.00000236/head//4),PushReplyOp(c21b678f/1000001aa8d.0000011c/head//4),PushReplyOp(87ab678f/1000002a2df.00000064/head//4),PushReplyOp(f06c678f/1000002bca6.0000013f/head//4),PushReplyOp(846d678f/1000002afc4.00000068/head//4)]) v2 -- ?+0 0x246e9a00 con 0x21468c60 -7039> 2014-11-08 18:14:47.692599 7fc5adb11700 5 -- op tracker -- seq: 46946, time: 2014-11-08 18:14:47.692599, event: done, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=0)) -7038> 2014-11-08 18:14:47.693306 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 315 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.690882) v2 ==== 47+0+0 (946077392 0 0) 0x22992b20 con 0x1eb9d6a0 -7037> 2014-11-08 18:14:47.693328 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.690882) v2 -- ?+0 0x22e5e720 con 0x1eb9d6a0 -7036> 2014-11-08 18:14:47.694935 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 315 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.690882) v2 ==== 47+0+0 (946077392 0 0) 0x20269860 con 0x1eb9c780 -7035> 2014-11-08 18:14:47.694950 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.690882) v2 -- ?+0 0x24880960 con 0x1eb9c780 -7034> 2014-11-08 18:14:47.717676 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.91s1 to 5734'8180 e5773) v1 -- ?+0 0x23c30780 con 0x21468c60 -7033> 2014-11-08 18:14:47.717726 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.91s1 5773 [PushReplyOp(853ff291/100000296ee.0000004b/head//4)]) v2 -- ?+0 0x1d5b3400 con 0x21468c60 -7032> 2014-11-08 18:14:47.798233 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 494 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.794818) v2 ==== 47+0+0 (1872569548 0 0) 0x24882d00 con 0x1eb9c8e0 -7031> 2014-11-08 18:14:47.798257 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.794818) v2 -- ?+0 0x20269860 con 0x1eb9c8e0 -7030> 2014-11-08 18:14:47.802243 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 494 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.794818) v2 ==== 47+0+0 (1872569548 0 0) 0x22e5c920 con 0x1eb9d800 -7029> 2014-11-08 18:14:47.802257 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.794818) v2 -- ?+0 0x22992b20 con 0x1eb9d800 -7028> 2014-11-08 18:14:47.847575 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3706 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.843867) v2 ==== 47+0+0 (2453452004 0 0) 0x22e9d280 con 0x1c9f8580 -7027> 2014-11-08 18:14:47.847619 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.843867) v2 -- ?+0 0x22e5c920 con 0x1c9f8580 -7026> 2014-11-08 18:14:47.847800 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3706 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:47.843867) v2 ==== 47+0+0 (2453452004 0 0) 0x24959fe0 con 0x1c8b7d20 -7025> 2014-11-08 18:14:47.847812 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:47.843867) v2 -- ?+0 0x24882d00 con 0x1c8b7d20 -7024> 2014-11-08 18:14:47.864331 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpReadReply(4.b2s1 5773 ECSubReadReply(tid=20434, attrs_read=0)) v1 -- ?+0 0x1c89e180 con 0x1c5c0580 -7023> 2014-11-08 18:14:47.864381 7fc5af314700 5 -- op tracker -- seq: 46933, time: 2014-11-08 18:14:47.864381, event: done, op: MOSDECSubOpRead(4.b2s2 5773 ECSubRead(tid=20434, to_read={c0e785b2/1000002b871.000000c1/head//4=0,4194304,a7e785b2/1000001b503.00000276/head//4=0,4194304,d15885b2/1000002a0cd.0000000a/head//4=0,4194304,14a885b2/1000001261a.00000000/head//4=0,4194304,28b885b2/100000297a9.0000008a/head//4=0,4194304}, attrs_to_read=)) -7022> 2014-11-08 18:14:48.011209 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1600 ==== pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) v3 ==== 757+0+0 (3976920021 0 0) 0x2366f980 con 0x21468c60 -7021> 2014-11-08 18:14:48.011247 7fc5a39d2700 5 -- op tracker -- seq: 46947, time: 2014-11-08 18:14:48.011103, event: header_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7020> 2014-11-08 18:14:48.011258 7fc5a39d2700 5 -- op tracker -- seq: 46947, time: 2014-11-08 18:14:48.011108, event: throttled, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7019> 2014-11-08 18:14:48.011261 7fc5a39d2700 5 -- op tracker -- seq: 46947, time: 2014-11-08 18:14:48.011201, event: all_read, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7018> 2014-11-08 18:14:48.011266 7fc5a39d2700 5 -- op tracker -- seq: 46947, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7017> 2014-11-08 18:14:48.011306 7fc5af314700 5 -- op tracker -- seq: 46947, time: 2014-11-08 18:14:48.011306, event: reached_pg, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7016> 2014-11-08 18:14:48.011319 7fc5af314700 5 -- op tracker -- seq: 46947, time: 2014-11-08 18:14:48.011319, event: started, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7015> 2014-11-08 18:14:48.011336 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1601 ==== MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (3353882026 0 0) 0x1ce67500 con 0x21468c60 -7014> 2014-11-08 18:14:48.011350 7fc5a39d2700 5 -- op tracker -- seq: 46948, time: 2014-11-08 18:14:48.011295, event: header_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) -7013> 2014-11-08 18:14:48.011357 7fc5a39d2700 5 -- op tracker -- seq: 46948, time: 2014-11-08 18:14:48.011296, event: throttled, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) -7012> 2014-11-08 18:14:48.011363 7fc5a39d2700 5 -- op tracker -- seq: 46948, time: 2014-11-08 18:14:48.011333, event: all_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) -7011> 2014-11-08 18:14:48.011368 7fc5a39d2700 5 -- op tracker -- seq: 46948, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) -7010> 2014-11-08 18:14:48.011383 7fc5af314700 5 -- op tracker -- seq: 46947, time: 2014-11-08 18:14:48.011383, event: done, op: pg_backfill(progress 4.8fs2 e 5773/5773 lb 846d678f/1000002afc4.00000068/head//4) -7009> 2014-11-08 18:14:48.011399 7fc5af314700 5 -- op tracker -- seq: 46948, time: 2014-11-08 18:14:48.011399, event: reached_pg, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) -7008> 2014-11-08 18:14:48.057381 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12054 ==== MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (4209920215 0 0) 0x2117d400 con 0x1c5c2680 -7007> 2014-11-08 18:14:48.057450 7fc5a6cfe700 5 -- op tracker -- seq: 46949, time: 2014-11-08 18:14:47.720169, event: header_read, op: MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7006> 2014-11-08 18:14:48.057465 7fc5a6cfe700 5 -- op tracker -- seq: 46949, time: 2014-11-08 18:14:47.720172, event: throttled, op: MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7005> 2014-11-08 18:14:48.057473 7fc5a6cfe700 5 -- op tracker -- seq: 46949, time: 2014-11-08 18:14:48.057370, event: all_read, op: MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7004> 2014-11-08 18:14:48.057481 7fc5a6cfe700 5 -- op tracker -- seq: 46949, time: 0.000000, event: dispatched, op: MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -7003> 2014-11-08 18:14:48.057549 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12055 ==== MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (3182667831 0 0) 0x230298c0 con 0x1c5c2680 -7002> 2014-11-08 18:14:48.057562 7fc5a6cfe700 5 -- op tracker -- seq: 46950, time: 2014-11-08 18:14:48.057515, event: header_read, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) -7001> 2014-11-08 18:14:48.057571 7fc5a6cfe700 5 -- op tracker -- seq: 46950, time: 2014-11-08 18:14:48.057516, event: throttled, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) -7000> 2014-11-08 18:14:48.057578 7fc5a6cfe700 5 -- op tracker -- seq: 46950, time: 2014-11-08 18:14:48.057548, event: all_read, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) -6999> 2014-11-08 18:14:48.057585 7fc5a6cfe700 5 -- op tracker -- seq: 46950, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) -6998> 2014-11-08 18:14:48.057796 7fc5ad310700 5 -- op tracker -- seq: 46950, time: 2014-11-08 18:14:48.057796, event: reached_pg, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) -6997> 2014-11-08 18:14:48.215755 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3736 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.215043) v2 ==== 47+0+0 (2464948686 0 0) 0x21227bc0 con 0x1cce3700 -6996> 2014-11-08 18:14:48.215783 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.215043) v2 -- ?+0 0x24959fe0 con 0x1cce3700 -6995> 2014-11-08 18:14:48.215802 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3736 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.215043) v2 ==== 47+0+0 (2464948686 0 0) 0x23681680 con 0x1cce3180 -6994> 2014-11-08 18:14:48.215812 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.215043) v2 -- ?+0 0x22e9d280 con 0x1cce3180 -6993> 2014-11-08 18:14:48.252108 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8520 ==== MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (861012318 0 0) 0x21612c00 con 0x1c5c0580 -6992> 2014-11-08 18:14:48.252151 7fc5a152e700 5 -- op tracker -- seq: 46951, time: 2014-11-08 18:14:48.101917, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6991> 2014-11-08 18:14:48.252170 7fc5a152e700 5 -- op tracker -- seq: 46951, time: 2014-11-08 18:14:48.101920, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6990> 2014-11-08 18:14:48.252184 7fc5a152e700 5 -- op tracker -- seq: 46951, time: 2014-11-08 18:14:48.252098, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6989> 2014-11-08 18:14:48.252197 7fc5a152e700 5 -- op tracker -- seq: 46951, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6988> 2014-11-08 18:14:48.252247 7fc5b0316700 5 -- op tracker -- seq: 46951, time: 2014-11-08 18:14:48.252247, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6987> 2014-11-08 18:14:48.252383 7fc5b0316700 5 -- op tracker -- seq: 46951, time: 2014-11-08 18:14:48.252383, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(954afcb5/10000019b1e.00000401/head//4, version: 397'1196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(954afcb5/10000019b1e.00000401/head//4@397'1196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e4bfcb5/1000001b50c.00000246/head//4, version: 3033'3283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e4bfcb5/1000001b50c.00000246/head//4@3033'3283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efdbfcb5/1000002ab79.00000023/head//4, version: 3468'5818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efdbfcb5/1000002ab79.00000023/head//4@3468'5818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6986> 2014-11-08 18:14:48.450571 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59249, attrs_read=0)) v1 -- ?+0 0x21a49b80 con 0x21468c60 -6985> 2014-11-08 18:14:48.455606 7fc5acb0f700 5 -- op tracker -- seq: 46945, time: 2014-11-08 18:14:48.455606, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59249, to_read={a1b1414e/1000002aa67.0000003f/head//4=0,4194304,fdb1414e/1000002952d.000000c8/head//4=0,4194304,3ce1414e/1000002a7f3.00000049/head//4=0,4194304,33a2414e/1000002b757.00000032/head//4=0,4194304,21d2414e/10000019d32.00000450/head//4=0,4194304}, attrs_to_read=)) -6984> 2014-11-08 18:14:48.455665 7fc5acb0f700 5 -- op tracker -- seq: 46949, time: 2014-11-08 18:14:48.455664, event: reached_pg, op: MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6983> 2014-11-08 18:14:48.455759 7fc5acb0f700 5 -- op tracker -- seq: 46949, time: 2014-11-08 18:14:48.455759, event: done, op: MOSDPGPush(4.67s1 5773 [PushOp(19b1ff67/1000001a111.0000003f/head//4, version: 397'3397, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(19b1ff67/1000001a111.0000003f/head//4@397'3397, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6982> 2014-11-08 18:14:48.468607 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x23c35460 con 0x1c5c0580 -6981> 2014-11-08 18:14:48.468652 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1d040960 con 0x1c5c0580 -6980> 2014-11-08 18:14:48.468659 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1d041a40 con 0x1c5c0580 -6979> 2014-11-08 18:14:48.468667 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(954afcb5/10000019b1e.00000401/head//4),PushReplyOp(9e4bfcb5/1000001b50c.00000246/head//4),PushReplyOp(efdbfcb5/1000002ab79.00000023/head//4)]) v2 -- ?+0 0x1d3a9400 con 0x1c5c0580 -6978> 2014-11-08 18:14:48.538744 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12056 ==== MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) v1 ==== 4195327+0+0 (994934170 0 0) 0x21a4bc00 con 0x1c5c2680 -6977> 2014-11-08 18:14:48.539353 7fc5a6cfe700 5 -- op tracker -- seq: 46952, time: 2014-11-08 18:14:48.302448, event: header_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) -6976> 2014-11-08 18:14:48.539368 7fc5a6cfe700 5 -- op tracker -- seq: 46952, time: 2014-11-08 18:14:48.302453, event: throttled, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) -6975> 2014-11-08 18:14:48.539377 7fc5a6cfe700 5 -- op tracker -- seq: 46952, time: 2014-11-08 18:14:48.538736, event: all_read, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) -6974> 2014-11-08 18:14:48.539385 7fc5a6cfe700 5 -- op tracker -- seq: 46952, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) -6973> 2014-11-08 18:14:48.539427 7fc5adb11700 5 -- op tracker -- seq: 46952, time: 2014-11-08 18:14:48.539427, event: reached_pg, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) -6972> 2014-11-08 18:14:48.570735 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_trim(4.67s1 to 5734'16521 e5773) v1 -- ?+0 0x1d044ec0 con 0x1c5c2680 -6971> 2014-11-08 18:14:48.591855 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDPGPush(4.15s0 5773 [PushOp(c958de15/10000028657.00000051/head//4, version: 3033'3848, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c958de15/10000028657.00000051/head//4@3033'3848, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2968de15/1000002a90f.00000070/head//4, version: 3458'5543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2968de15/1000002a90f.00000070/head//4@3458'5543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x21610000 con 0x1c5c2680 -6970> 2014-11-08 18:14:48.591917 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPush(4.15s2 5773 [PushOp(c958de15/10000028657.00000051/head//4, version: 3033'3848, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c958de15/10000028657.00000051/head//4@3033'3848, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2968de15/1000002a90f.00000070/head//4, version: 3458'5543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2968de15/1000002a90f.00000070/head//4@3458'5543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress( !first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x24847200 con 0x21468c60 -6969> 2014-11-08 18:14:48.591938 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- MOSDPGPush(4.15s1 5773 [PushOp(c958de15/10000028657.00000051/head//4, version: 3033'3848, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c958de15/10000028657.00000051/head//4@3033'3848, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2968de15/1000002a90f.00000070/head//4, version: 3458'5543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2968de15/1000002a90f.00000070/head//4@3458'5543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1c6a7c00 con 0x1c7911e0 -6968> 2014-11-08 18:14:48.606759 7fc5adb11700 5 -- op tracker -- seq: 46952, time: 2014-11-08 18:14:48.606758, event: done, op: MOSDECSubOpReadReply(4.15s1 5773 ECSubReadReply(tid=4408, attrs_read=2)) -6967> 2014-11-08 18:14:48.675715 7fc5c362f700 5 osd.9 5773 tick -6966> 2014-11-08 18:14:48.716270 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3737 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.715754) v2 ==== 47+0+0 (746554481 0 0) 0x23c37da0 con 0x1cce3180 -6965> 2014-11-08 18:14:48.716304 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.715754) v2 -- ?+0 0x23681680 con 0x1cce3180 -6964> 2014-11-08 18:14:48.716338 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3737 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.715754) v2 ==== 47+0+0 (746554481 0 0) 0x1ec26720 con 0x1cce3700 -6963> 2014-11-08 18:14:48.716352 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.715754) v2 -- ?+0 0x21227bc0 con 0x1cce3700 -6962> 2014-11-08 18:14:48.744351 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist [0,0,0,0,0,0,0,0,0,0,2]) -6961> 2014-11-08 18:14:48.744394 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x1dbfdbe0 con 0x1cae2940 -6960> 2014-11-08 18:14:48.744422 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x2286f9e0 con 0x1cae2680 -6959> 2014-11-08 18:14:48.744433 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x2286ecc0 con 0x1c5c02c0 -6958> 2014-11-08 18:14:48.744444 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x23922580 con 0x1cae0160 -6957> 2014-11-08 18:14:48.744454 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x2286a760 con 0x1cae0f20 -6956> 2014-11-08 18:14:48.744474 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x2286e360 con 0x1cae0c60 -6955> 2014-11-08 18:14:48.744485 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x213bc560 con 0x1cae23c0 -6954> 2014-11-08 18:14:48.744495 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x213bb2a0 con 0x1cae56a0 -6953> 2014-11-08 18:14:48.744505 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x2127f9e0 con 0x1cbf9ce0 -6952> 2014-11-08 18:14:48.744516 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x24881e00 con 0x1d603020 -6951> 2014-11-08 18:14:48.744526 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x2127d820 con 0x1dbd9b80 -6950> 2014-11-08 18:14:48.744536 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x1dbfcb00 con 0x1d235d80 -6949> 2014-11-08 18:14:48.744547 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x25e84920 con 0x1cae1fa0 -6948> 2014-11-08 18:14:48.744557 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x1deccb00 con 0x1cae1ce0 -6947> 2014-11-08 18:14:48.744566 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x1e533660 con 0x1cae1340 -6946> 2014-11-08 18:14:48.744588 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x229090e0 con 0x1cae1080 -6945> 2014-11-08 18:14:48.744597 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x257c3a20 con 0x1cae1a20 -6944> 2014-11-08 18:14:48.744607 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:48.744391) v2 -- ?+0 0x22349680 con 0x1cae2c00 -6943> 2014-11-08 18:14:48.745265 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x1f4c03c0 con 0x1cae2680 -6942> 2014-11-08 18:14:48.745289 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x249594a0 con 0x1cae23c0 -6941> 2014-11-08 18:14:48.745299 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x25838960 con 0x1cae56a0 -6940> 2014-11-08 18:14:48.745456 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x2290b0c0 con 0x1cae0160 -6939> 2014-11-08 18:14:48.745468 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x1e8921c0 con 0x1cae0c60 -6938> 2014-11-08 18:14:48.745477 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 308 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x1db1f260 con 0x1dbd9b80 -6937> 2014-11-08 18:14:48.745502 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 308 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x22455820 con 0x1d235d80 -6936> 2014-11-08 18:14:48.745680 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x25e81860 con 0x1cae1340 -6935> 2014-11-08 18:14:48.745848 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x236814a0 con 0x1cae2940 -6934> 2014-11-08 18:14:48.746001 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x211e7080 con 0x1cae0f20 -6933> 2014-11-08 18:14:48.746016 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x1eacab20 con 0x1cae1fa0 -6932> 2014-11-08 18:14:48.746042 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x4537bc0 con 0x1cae1ce0 -6931> 2014-11-08 18:14:48.746053 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x1d323fc0 con 0x1cae1a20 -6930> 2014-11-08 18:14:48.746063 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x2127d280 con 0x1cae2c00 -6929> 2014-11-08 18:14:48.746071 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 501 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x2234bde0 con 0x1d603020 -6928> 2014-11-08 18:14:48.746080 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x257c2b20 con 0x1c5c02c0 -6927> 2014-11-08 18:14:48.747738 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3721 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x1deca3a0 con 0x1cae1080 -6926> 2014-11-08 18:14:48.750255 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 501 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.744391) v2 ==== 47+0+0 (3462903341 0 0) 0x246c9fe0 con 0x1cbf9ce0 -6925> 2014-11-08 18:14:48.780510 7fc5a61fa700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 330 ==== MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) v2 ==== 175+0+0 (3727099216 0 0) 0x1c6a7c00 con 0x1c7911e0 -6924> 2014-11-08 18:14:48.780539 7fc5a61fa700 5 -- op tracker -- seq: 46953, time: 2014-11-08 18:14:48.780454, event: header_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6923> 2014-11-08 18:14:48.780548 7fc5a61fa700 5 -- op tracker -- seq: 46953, time: 2014-11-08 18:14:48.780455, event: throttled, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6922> 2014-11-08 18:14:48.780552 7fc5a61fa700 5 -- op tracker -- seq: 46953, time: 2014-11-08 18:14:48.780509, event: all_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6921> 2014-11-08 18:14:48.780555 7fc5a61fa700 5 -- op tracker -- seq: 46953, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6920> 2014-11-08 18:14:48.780586 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 328 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x22450d20 con 0x1c7911e0 -6919> 2014-11-08 18:14:48.780595 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780361, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -6918> 2014-11-08 18:14:48.780598 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780363, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -6917> 2014-11-08 18:14:48.780601 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780415, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -6916> 2014-11-08 18:14:48.780603 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780593, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -6915> 2014-11-08 18:14:48.780606 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780606, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -6914> 2014-11-08 18:14:48.780612 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780612, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -6913> 2014-11-08 18:14:48.780617 7fc5b9b29700 5 -- op tracker -- seq: 46954, time: 2014-11-08 18:14:48.780617, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -6912> 2014-11-08 18:14:48.780627 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 329 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x22452760 con 0x1c7911e0 -6911> 2014-11-08 18:14:48.780632 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780431, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -6910> 2014-11-08 18:14:48.780635 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780431, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -6909> 2014-11-08 18:14:48.780637 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780450, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -6908> 2014-11-08 18:14:48.780640 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780632, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -6907> 2014-11-08 18:14:48.780642 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780642, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -6906> 2014-11-08 18:14:48.780646 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780646, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -6905> 2014-11-08 18:14:48.780649 7fc5b9b29700 5 -- op tracker -- seq: 46955, time: 2014-11-08 18:14:48.780649, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -6904> 2014-11-08 18:14:48.780666 7fc5b0316700 5 -- op tracker -- seq: 46953, time: 2014-11-08 18:14:48.780666, event: reached_pg, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6903> 2014-11-08 18:14:48.780705 7fc5b0316700 5 -- op tracker -- seq: 46953, time: 2014-11-08 18:14:48.780705, event: done, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6902> 2014-11-08 18:14:48.802928 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3674 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.800386) v2 ==== 47+0+0 (2618369172 0 0) 0x231430c0 con 0x1cce3440 -6901> 2014-11-08 18:14:48.802979 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.800386) v2 -- ?+0 0x1ec26720 con 0x1cce3440 -6900> 2014-11-08 18:14:48.803022 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3674 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.800386) v2 ==== 47+0+0 (2618369172 0 0) 0x224a5640 con 0x1cce2ec0 -6899> 2014-11-08 18:14:48.803033 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.800386) v2 -- ?+0 0x23c37da0 con 0x1cce2ec0 -6898> 2014-11-08 18:14:48.816016 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8521 ==== MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67076b16/ 1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (1946303454 0 0) 0x1d3a9400 con 0x1c5c0580 -6897> 2014-11-08 18:14:48.816080 7fc5a152e700 5 -- op tracker -- seq: 46956, time: 2014-11-08 18:14:48.531518, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67076b16/1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6896> 2014-11-08 18:14:48.816110 7fc5a152e700 5 -- op tracker -- seq: 46956, time: 2014-11-08 18:14:48.531522, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 67076b16/1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6895> 2014-11-08 18:14:48.816135 7fc5a152e700 5 -- op tracker -- seq: 46956, time: 2014-11-08 18:14:48.813481, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67076b16/ 1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6894> 2014-11-08 18:14:48.816160 7fc5a152e700 5 -- op tracker -- seq: 46956, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67076b16/ 1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6893> 2014-11-08 18:14:48.816218 7fc5afb15700 5 -- op tracker -- seq: 46956, time: 2014-11-08 18:14:48.816218, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67076b16/1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6892> 2014-11-08 18:14:48.816412 7fc5a152e700 5 -- op tracker -- seq: 46956, time: 2014-11-08 18:14:48.816412, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(33a36b16/1000002b635.000000ce/head//4, version: 3473'6741, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a36b16/1000002b635.000000ce/head//4@3473'6741, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(95046b16/1000001a643.000000e8/head//4, version: 397'1994, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(95046b16/1000001a643.000000e8/head//4@397'1994, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(efb66b16/100000294ee.00000042/head//4, version: 3033'4056, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(efb66b16/100000294ee.00000042/head//4@3033'4056, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(67076b16/1000002b9c9.00000052/head//4, version: 3473'7243, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67076b16/ 1000002b9c9.00000052/head//4@3473'7243, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(48276b16/1000002bca6.0000002c/head//4, version: 3485'7656, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(48276b16/1000002bca6.0000002c/head//4@3485'7656, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6891> 2014-11-08 18:14:48.898324 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 495 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.895806) v2 ==== 47+0+0 (1474172562 0 0) 0x22e58000 con 0x1eb9d800 -6890> 2014-11-08 18:14:48.898366 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.895806) v2 -- ?+0 0x224a5640 con 0x1eb9d800 -6889> 2014-11-08 18:14:48.898521 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 495 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.895806) v2 ==== 47+0+0 (1474172562 0 0) 0x24885280 con 0x1eb9c8e0 -6888> 2014-11-08 18:14:48.898535 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.895806) v2 -- ?+0 0x231430c0 con 0x1eb9c8e0 -6887> 2014-11-08 18:14:48.909994 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1602 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x22452940 con 0x21468c60 -6886> 2014-11-08 18:14:48.910032 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.909648, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -6885> 2014-11-08 18:14:48.910040 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.909651, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -6884> 2014-11-08 18:14:48.910043 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.909752, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -6883> 2014-11-08 18:14:48.910045 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.910026, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -6882> 2014-11-08 18:14:48.910048 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.910048, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -6881> 2014-11-08 18:14:48.910056 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.910056, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -6880> 2014-11-08 18:14:48.910062 7fc5b9b29700 5 -- op tracker -- seq: 46957, time: 2014-11-08 18:14:48.910062, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -6879> 2014-11-08 18:14:48.910072 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1603 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x22452d00 con 0x21468c60 -6878> 2014-11-08 18:14:48.910078 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.909769, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -6877> 2014-11-08 18:14:48.910081 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.909770, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -6876> 2014-11-08 18:14:48.910084 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.909806, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -6875> 2014-11-08 18:14:48.910086 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.910078, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -6874> 2014-11-08 18:14:48.910089 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.910089, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -6873> 2014-11-08 18:14:48.910092 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.910092, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -6872> 2014-11-08 18:14:48.910095 7fc5b9b29700 5 -- op tracker -- seq: 46958, time: 2014-11-08 18:14:48.910095, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -6871> 2014-11-08 18:14:48.949496 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3707 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.946723) v2 ==== 47+0+0 (472743876 0 0) 0x22e9c380 con 0x1c9f8580 -6870> 2014-11-08 18:14:48.949540 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.946723) v2 -- ?+0 0x22e58000 con 0x1c9f8580 -6869> 2014-11-08 18:14:48.950405 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3707 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:48.946723) v2 ==== 47+0+0 (472743876 0 0) 0x1d8b90e0 con 0x1c8b7d20 -6868> 2014-11-08 18:14:48.950418 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:48.946723) v2 -- ?+0 0x24885280 con 0x1c8b7d20 -6867> 2014-11-08 18:14:48.969018 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d042940 con 0x1c5c0580 -6866> 2014-11-08 18:14:48.969101 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x4535280 con 0x1c5c0580 -6865> 2014-11-08 18:14:48.969109 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1deccce0 con 0x1c5c0580 -6864> 2014-11-08 18:14:48.969116 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1decc560 con 0x1c5c0580 -6863> 2014-11-08 18:14:48.969122 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x202690e0 con 0x1c5c0580 -6862> 2014-11-08 18:14:49.085472 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12057 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x1d044ec0 con 0x1c5c2680 -6861> 2014-11-08 18:14:49.085494 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085379, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -6860> 2014-11-08 18:14:49.085499 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085382, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -6859> 2014-11-08 18:14:49.085502 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085427, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -6858> 2014-11-08 18:14:49.085504 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085490, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -6857> 2014-11-08 18:14:49.085507 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085507, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -6856> 2014-11-08 18:14:49.085513 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085512, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -6855> 2014-11-08 18:14:49.085517 7fc5b9b29700 5 -- op tracker -- seq: 46959, time: 2014-11-08 18:14:49.085517, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -6854> 2014-11-08 18:14:49.085526 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12058 ==== pg_trim(4.15 to 5734'8279 e5773) v1 ==== 34+0+0 (2055289816 0 0) 0x1d072b20 con 0x1c5c2680 -6853> 2014-11-08 18:14:49.085532 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085439, event: header_read, op: pg_trim(4.15 to 5734'8279 e5773) -6852> 2014-11-08 18:14:49.085535 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085440, event: throttled, op: pg_trim(4.15 to 5734'8279 e5773) -6851> 2014-11-08 18:14:49.085537 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085459, event: all_read, op: pg_trim(4.15 to 5734'8279 e5773) -6850> 2014-11-08 18:14:49.085539 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085531, event: dispatched, op: pg_trim(4.15 to 5734'8279 e5773) -6849> 2014-11-08 18:14:49.085542 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085542, event: waiting_for_osdmap, op: pg_trim(4.15 to 5734'8279 e5773) -6848> 2014-11-08 18:14:49.085546 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085546, event: started, op: pg_trim(4.15 to 5734'8279 e5773) -6847> 2014-11-08 18:14:49.085549 7fc5b9b29700 5 -- op tracker -- seq: 46960, time: 2014-11-08 18:14:49.085549, event: done, op: pg_trim(4.15 to 5734'8279 e5773) -6846> 2014-11-08 18:14:49.154554 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDPGPushReply(4.67s1 5773 [PushReplyOp(19b1ff67/1000001a111.0000003f/head//4)]) v2 -- ?+0 0x1d3a8000 con 0x1c5c2680 -6845> 2014-11-08 18:14:49.164297 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(33a36b16/1000002b635.000000ce/head//4),PushReplyOp(95046b16/1000001a643.000000e8/head//4),PushReplyOp(efb66b16/100000294ee.00000042/head//4),PushReplyOp(67076b16/1000002b9c9.00000052/head//4),PushReplyOp(48276b16/1000002bca6.0000002c/head//4)]) v2 -- ?+0 0x1ebb5c00 con 0x1c5c0580 -6844> 2014-11-08 18:14:49.166725 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8522 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) v3 ==== 757+0+0 (793129094 0 0) 0x1e58ad00 con 0x1c5c0580 -6843> 2014-11-08 18:14:49.166759 7fc5a152e700 5 -- op tracker -- seq: 46961, time: 2014-11-08 18:14:49.166534, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6842> 2014-11-08 18:14:49.166774 7fc5a152e700 5 -- op tracker -- seq: 46961, time: 2014-11-08 18:14:49.166536, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6841> 2014-11-08 18:14:49.166784 7fc5a152e700 5 -- op tracker -- seq: 46961, time: 2014-11-08 18:14:49.166719, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6840> 2014-11-08 18:14:49.166793 7fc5a152e700 5 -- op tracker -- seq: 46961, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6839> 2014-11-08 18:14:49.167897 7fc5afb15700 5 -- op tracker -- seq: 46961, time: 2014-11-08 18:14:49.167897, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6838> 2014-11-08 18:14:49.167909 7fc5afb15700 5 -- op tracker -- seq: 46961, time: 2014-11-08 18:14:49.167909, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6837> 2014-11-08 18:14:49.167968 7fc5afb15700 5 -- op tracker -- seq: 46961, time: 2014-11-08 18:14:49.167968, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 48276b16/1000002bca6.0000002c/head//4) -6836> 2014-11-08 18:14:49.381963 7fc5af314700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.8fs0 5773 ECSubReadReply(tid=59250, attrs_read=0)) v1 -- ?+0 0x1c89dc80 con 0x21468c60 -6835> 2014-11-08 18:14:49.386706 7fc5af314700 5 -- op tracker -- seq: 46948, time: 2014-11-08 18:14:49.386706, event: done, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59250, to_read={49d678f/1000002bd18.00000008/head//4=0,4194304,d6bd678f/1000002b98a.0000007c/head//4=0,4194304,f87e678f/1000001b237.0000000b/head//4=0,4194304,774f678f/1000002ab29.00000033/head//4=0,4194304,c5ef678f/1000002b09a.00000086/head//4=0,4194304}, attrs_to_read=)) -6834> 2014-11-08 18:14:49.480374 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.57s0 5773 ECSubReadReply(tid=25188, attrs_read=0)) v1 -- ?+0 0x2183b980 con 0x1c5c2680 -6833> 2014-11-08 18:14:49.480412 7fc5ad310700 5 -- op tracker -- seq: 46950, time: 2014-11-08 18:14:49.480412, event: done, op: MOSDECSubOpRead(4.57s1 5773 ECSubRead(tid=25188, to_read={8e270857/1000001b53f.00000477/head//4=0,4194304,b9a80857/1000001a0af.00000342/head//4=0,4194304,9bd80857/10000029560.0000001f/head//4=0,4194304,72990857/10000028606.00000030/head//4=0,4194304,ab6b0857/10000019a09.00000450/head//4=0,4194304}, attrs_to_read=)) -6832> 2014-11-08 18:14:49.581845 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8523 ==== MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098072+0+0 (481856120 0 0) 0x1ebb5c00 con 0x1c5c0580 -6831> 2014-11-08 18:14:49.581887 7fc5a152e700 5 -- op tracker -- seq: 46962, time: 2014-11-08 18:14:49.425556, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6830> 2014-11-08 18:14:49.581899 7fc5a152e700 5 -- op tracker -- seq: 46962, time: 2014-11-08 18:14:49.425561, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6829> 2014-11-08 18:14:49.581905 7fc5a152e700 5 -- op tracker -- seq: 46962, time: 2014-11-08 18:14:49.581835, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6828> 2014-11-08 18:14:49.581911 7fc5a152e700 5 -- op tracker -- seq: 46962, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6827> 2014-11-08 18:14:49.582018 7fc5afb15700 5 -- op tracker -- seq: 46962, time: 2014-11-08 18:14:49.582018, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6826> 2014-11-08 18:14:49.582098 7fc5afb15700 5 -- op tracker -- seq: 46962, time: 2014-11-08 18:14:49.582098, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(ca876b16/1000002bd4d.0000001f/head//4, version: 4864'8260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ca876b16/1000002bd4d.0000001f/head//4@4864'8260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6825> 2014-11-08 18:14:49.660640 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x20268780 con 0x1c5c0580 -6824> 2014-11-08 18:14:49.660829 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(ca876b16/1000002bd4d.0000001f/head//4)]) v2 -- ?+0 0x21611a00 con 0x1c5c0580 -6823> 2014-11-08 18:14:49.666968 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8524 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) v3 ==== 757+0+0 (300884555 0 0) 0x226f1f80 con 0x1c5c0580 -6822> 2014-11-08 18:14:49.666997 7fc5a152e700 5 -- op tracker -- seq: 46963, time: 2014-11-08 18:14:49.666873, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6821> 2014-11-08 18:14:49.667007 7fc5a152e700 5 -- op tracker -- seq: 46963, time: 2014-11-08 18:14:49.666876, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6820> 2014-11-08 18:14:49.667012 7fc5a152e700 5 -- op tracker -- seq: 46963, time: 2014-11-08 18:14:49.666962, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6819> 2014-11-08 18:14:49.667015 7fc5a152e700 5 -- op tracker -- seq: 46963, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6818> 2014-11-08 18:14:49.667054 7fc5ad310700 5 -- op tracker -- seq: 46963, time: 2014-11-08 18:14:49.667053, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6817> 2014-11-08 18:14:49.667066 7fc5ad310700 5 -- op tracker -- seq: 46963, time: 2014-11-08 18:14:49.667066, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6816> 2014-11-08 18:14:49.667118 7fc5ad310700 5 -- op tracker -- seq: 46963, time: 2014-11-08 18:14:49.667118, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ca876b16/1000002bd4d.0000001f/head//4) -6815> 2014-11-08 18:14:49.675880 7fc5c362f700 5 osd.9 5773 tick -6814> 2014-11-08 18:14:49.922036 7fc5b2b1b700 10 monclient: tick -6813> 2014-11-08 18:14:49.922059 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:14:19.922057) -6812> 2014-11-08 18:14:49.922077 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:14:49.922077; renew after: 2014-11-08 18:16:29.290049) -- no -6811> 2014-11-08 18:14:50.151531 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1604 ==== MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(774f678f/ 1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490148+0+0 (2418493230 0 0) 0x24847200 con 0x21468c60 -6810> 2014-11-08 18:14:50.151603 7fc5a39d2700 5 -- op tracker -- seq: 46964, time: 2014-11-08 18:14:49.925919, event: header_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(774f678f/1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6809> 2014-11-08 18:14:50.151627 7fc5a39d2700 5 -- op tracker -- seq: 46964, time: 2014-11-08 18:14:49.925923, event: throttled, op: MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 774f678f/1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6808> 2014-11-08 18:14:50.151647 7fc5a39d2700 5 -- op tracker -- seq: 46964, time: 2014-11-08 18:14:50.151518, event: all_read, op: MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(774f678f/ 1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6807> 2014-11-08 18:14:50.151667 7fc5a39d2700 5 -- op tracker -- seq: 46964, time: 0.000000, event: dispatched, op: MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(774f678f/ 1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6806> 2014-11-08 18:14:50.151725 7fc5acb0f700 5 -- op tracker -- seq: 46964, time: 2014-11-08 18:14:50.151725, event: reached_pg, op: MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(774f678f/1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6805> 2014-11-08 18:14:50.151912 7fc5acb0f700 5 -- op tracker -- seq: 46964, time: 2014-11-08 18:14:50.151912, event: done, op: MOSDPGPush(4.8fs2 5773 [PushOp(49d678f/1000002bd18.00000008/head//4, version: 4864'7982, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(49d678f/1000002bd18.00000008/head//4@4864'7982, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6bd678f/1000002b98a.0000007c/head//4, version: 3473'6954, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6bd678f/1000002b98a.0000007c/head//4@3473'6954, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f87e678f/1000001b237.0000000b/head//4, version: 2829'2601, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f87e678f/1000001b237.0000000b/head//4@2829'2601, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(774f678f/1000002ab29.00000033/head//4, version: 3467'5650, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(774f678f/ 1000002ab29.00000033/head//4@3467'5650, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5ef678f/1000002b09a.00000086/head//4, version: 4157'7705, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5ef678f/1000002b09a.00000086/head//4@4157'7705, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6804> 2014-11-08 18:14:50.358703 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x202692c0 con 0x21468c60 -6803> 2014-11-08 18:14:50.358746 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x2026cb00 con 0x21468c60 -6802> 2014-11-08 18:14:50.358753 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x2026d820 con 0x21468c60 -6801> 2014-11-08 18:14:50.358799 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x20268b40 con 0x21468c60 -6800> 2014-11-08 18:14:50.358805 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_trim(4.8fs0 to 5734'8119 e5773) v1 -- ?+0 0x20269fe0 con 0x21468c60 -6799> 2014-11-08 18:14:50.358813 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPushReply(4.8fs0 5773 [PushReplyOp(49d678f/1000002bd18.00000008/head//4),PushReplyOp(d6bd678f/1000002b98a.0000007c/head//4),PushReplyOp(f87e678f/1000001b237.0000000b/head//4),PushReplyOp(774f678f/1000002ab29.00000033/head//4),PushReplyOp(c5ef678f/1000002b09a.00000086/head//4)]) v2 -- ?+0 0x2117d400 con 0x21468c60 -6798> 2014-11-08 18:14:50.639256 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8525 ==== MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8735004+0+0 (398257701 0 0) 0x21611a00 con 0x1c5c0580 -6797> 2014-11-08 18:14:50.639313 7fc5a152e700 5 -- op tracker -- seq: 46965, time: 2014-11-08 18:14:50.484454, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6796> 2014-11-08 18:14:50.639339 7fc5a152e700 5 -- op tracker -- seq: 46965, time: 2014-11-08 18:14:50.484459, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6795> 2014-11-08 18:14:50.639358 7fc5a152e700 5 -- op tracker -- seq: 46965, time: 2014-11-08 18:14:50.639245, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6794> 2014-11-08 18:14:50.639377 7fc5a152e700 5 -- op tracker -- seq: 46965, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6793> 2014-11-08 18:14:50.639431 7fc5afb15700 5 -- op tracker -- seq: 46965, time: 2014-11-08 18:14:50.639431, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6792> 2014-11-08 18:14:50.639608 7fc5afb15700 5 -- op tracker -- seq: 46965, time: 2014-11-08 18:14:50.639608, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(62586b16/1000001b629.0000000a/head//4, version: 3033'3339, data_included: [0~342016], data_size: 342016, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(62586b16/1000001b629.0000000a/head//4@3033'3339, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:684032, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4a86b16/1000002af81.0000002d/head//4, version: 4157'7800, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4a86b16/1000002af81.0000002d/head//4@4157'7800, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e9a6b16/1000002a338.00000031/head//4, version: 3033'4818, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e9a6b16/1000002a338.00000031/head//4@3033'4818, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ada6b16/1000002b067.00000038/head//4, version: 4157'7935, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ada6b16/ 1000002b067.00000038/head//4@4157'7935, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cacb6b16/1000002ac72.00000037/head//4, version: 3468'5927, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cacb6b16/1000002ac72.00000037/head//4@3468'5927, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6791> 2014-11-08 18:14:50.676028 7fc5c362f700 5 osd.9 5773 tick -6790> 2014-11-08 18:14:50.676074 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -6789> 2014-11-08 18:14:50.676084 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(2 pgs tid 1328 v 0) v1 -- ?+0 0x248ad800 con 0x44ed6a0 -6788> 2014-11-08 18:14:50.819439 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2026c560 con 0x1c5c0580 -6787> 2014-11-08 18:14:50.819475 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2026fbc0 con 0x1c5c0580 -6786> 2014-11-08 18:14:50.819482 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ee360 con 0x1c5c0580 -6785> 2014-11-08 18:14:50.819487 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239eba20 con 0x1c5c0580 -6784> 2014-11-08 18:14:50.819493 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ed280 con 0x1c5c0580 -6783> 2014-11-08 18:14:51.012282 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1503 ==== pg_stats_ack(2 pgs tid 1328) v1 ==== 62+0+0 (358192509 0 0) 0x2286c560 con 0x44ed6a0 -6782> 2014-11-08 18:14:51.045044 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -6781> 2014-11-08 18:14:51.045079 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x246ce720 con 0x1cae2940 -6780> 2014-11-08 18:14:51.045105 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x4530000 con 0x1cae2680 -6779> 2014-11-08 18:14:51.045114 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1e8912c0 con 0x1c5c02c0 -6778> 2014-11-08 18:14:51.045124 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1eacc560 con 0x1cae0160 -6777> 2014-11-08 18:14:51.045133 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x22d812c0 con 0x1cae0f20 -6776> 2014-11-08 18:14:51.045144 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1db1e720 con 0x1cae0c60 -6775> 2014-11-08 18:14:51.045152 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x2583ecc0 con 0x1cae23c0 -6774> 2014-11-08 18:14:51.045162 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x24959c20 con 0x1cae56a0 -6773> 2014-11-08 18:14:51.045171 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1f4c6180 con 0x1cbf9ce0 -6772> 2014-11-08 18:14:51.045180 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x23687800 con 0x1d603020 -6771> 2014-11-08 18:14:51.045190 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1dbfb480 con 0x1dbd9b80 -6770> 2014-11-08 18:14:51.045199 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1d3201e0 con 0x1d235d80 -6769> 2014-11-08 18:14:51.045209 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1decd280 con 0x1cae1fa0 -6768> 2014-11-08 18:14:51.045217 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x25e821c0 con 0x1cae1ce0 -6767> 2014-11-08 18:14:51.045226 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x22d87bc0 con 0x1cae1340 -6766> 2014-11-08 18:14:51.045236 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1db1dbe0 con 0x1cae1080 -6765> 2014-11-08 18:14:51.045244 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x1dbfdfa0 con 0x1cae1a20 -6764> 2014-11-08 18:14:51.045254 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:51.045075) v2 -- ?+0 0x246caee0 con 0x1cae2c00 -6763> 2014-11-08 18:14:51.045877 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x24958d20 con 0x1cae23c0 -6762> 2014-11-08 18:14:51.045897 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x25e83840 con 0x1cae1340 -6761> 2014-11-08 18:14:51.046338 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x1eacecc0 con 0x1cae1fa0 -6760> 2014-11-08 18:14:51.046354 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x23680780 con 0x1cae2940 -6759> 2014-11-08 18:14:51.046364 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x1f4c2580 con 0x1cae2680 -6758> 2014-11-08 18:14:51.046373 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x2583c560 con 0x1cae56a0 -6757> 2014-11-08 18:14:51.046381 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 309 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x2583f620 con 0x1dbd9b80 -6756> 2014-11-08 18:14:51.046390 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x2290ad00 con 0x1cae0160 -6755> 2014-11-08 18:14:51.046399 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x1e6aa940 con 0x1cae0f20 -6754> 2014-11-08 18:14:51.046408 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x1e897bc0 con 0x1cae0c60 -6753> 2014-11-08 18:14:51.047044 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x2127f260 con 0x1cae2c00 -6752> 2014-11-08 18:14:51.047487 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x1d326540 con 0x1cae1a20 -6751> 2014-11-08 18:14:51.047503 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 502 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x22349c20 con 0x1d603020 -6750> 2014-11-08 18:14:51.047514 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 309 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x22579e00 con 0x1d235d80 -6749> 2014-11-08 18:14:51.047523 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x1d047440 con 0x1cae1080 -6748> 2014-11-08 18:14:51.047531 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x4537080 con 0x1cae1ce0 -6747> 2014-11-08 18:14:51.048917 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3722 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x257c7800 con 0x1c5c02c0 -6746> 2014-11-08 18:14:51.048936 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 502 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.045075) v2 ==== 47+0+0 (2806481847 0 0) 0x246cf440 con 0x1cbf9ce0 -6745> 2014-11-08 18:14:51.103400 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3675 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:51.101007) v2 ==== 47+0+0 (3991406231 0 0) 0x23144920 con 0x1cce3440 -6744> 2014-11-08 18:14:51.103400 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3675 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:51.101007) v2 ==== 47+0+0 (3991406231 0 0) 0x224a32a0 con 0x1cce2ec0 -6743> 2014-11-08 18:14:51.103451 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.101007) v2 -- ?+0 0x1d8b90e0 con 0x1cce3440 -6742> 2014-11-08 18:14:51.103518 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:51.101007) v2 -- ?+0 0x22e9c380 con 0x1cce2ec0 -6741> 2014-11-08 18:14:51.676171 7fc5c362f700 5 osd.9 5773 tick -6740> 2014-11-08 18:14:51.719171 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12059 ==== MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) v2 ==== 175+0+0 (2702720310 0 0) 0x1d3a8000 con 0x1c5c2680 -6739> 2014-11-08 18:14:51.719207 7fc5a6cfe700 5 -- op tracker -- seq: 46966, time: 2014-11-08 18:14:51.719070, event: header_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6738> 2014-11-08 18:14:51.719216 7fc5a6cfe700 5 -- op tracker -- seq: 46966, time: 2014-11-08 18:14:51.719075, event: throttled, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6737> 2014-11-08 18:14:51.719219 7fc5a6cfe700 5 -- op tracker -- seq: 46966, time: 2014-11-08 18:14:51.719162, event: all_read, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6736> 2014-11-08 18:14:51.719222 7fc5a6cfe700 5 -- op tracker -- seq: 46966, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6735> 2014-11-08 18:14:51.719276 7fc5adb11700 5 -- op tracker -- seq: 46966, time: 2014-11-08 18:14:51.719276, event: reached_pg, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6734> 2014-11-08 18:14:51.719315 7fc5adb11700 5 -- op tracker -- seq: 46966, time: 2014-11-08 18:14:51.719315, event: done, op: MOSDPGPushReply(4.15s1 5773 [PushReplyOp(c958de15/10000028657.00000051/head//4),PushReplyOp(2968de15/1000002a90f.00000070/head//4)]) -6733> 2014-11-08 18:14:52.149437 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8526 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) v3 ==== 757+0+0 (3911181953 0 0) 0x2366ba80 con 0x1c5c0580 -6732> 2014-11-08 18:14:52.149472 7fc5a152e700 5 -- op tracker -- seq: 46967, time: 2014-11-08 18:14:52.149333, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6731> 2014-11-08 18:14:52.149480 7fc5a152e700 5 -- op tracker -- seq: 46967, time: 2014-11-08 18:14:52.149338, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6730> 2014-11-08 18:14:52.149483 7fc5a152e700 5 -- op tracker -- seq: 46967, time: 2014-11-08 18:14:52.149428, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6729> 2014-11-08 18:14:52.149486 7fc5a152e700 5 -- op tracker -- seq: 46967, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6728> 2014-11-08 18:14:52.149549 7fc5b0316700 5 -- op tracker -- seq: 46967, time: 2014-11-08 18:14:52.149549, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6727> 2014-11-08 18:14:52.149560 7fc5b0316700 5 -- op tracker -- seq: 46967, time: 2014-11-08 18:14:52.149560, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6726> 2014-11-08 18:14:52.149612 7fc5b0316700 5 -- op tracker -- seq: 46967, time: 2014-11-08 18:14:52.149612, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb efdbfcb5/1000002ab79.00000023/head//4) -6725> 2014-11-08 18:14:52.432627 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8527 ==== MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2029892839 0 0) 0x1d3a9400 con 0x1c5c0580 -6724> 2014-11-08 18:14:52.432662 7fc5a152e700 5 -- op tracker -- seq: 46968, time: 2014-11-08 18:14:52.389205, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6723> 2014-11-08 18:14:52.432675 7fc5a152e700 5 -- op tracker -- seq: 46968, time: 2014-11-08 18:14:52.389209, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6722> 2014-11-08 18:14:52.432682 7fc5a152e700 5 -- op tracker -- seq: 46968, time: 2014-11-08 18:14:52.432620, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6721> 2014-11-08 18:14:52.432696 7fc5a152e700 5 -- op tracker -- seq: 46968, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6720> 2014-11-08 18:14:52.432734 7fc5adb11700 5 -- op tracker -- seq: 46968, time: 2014-11-08 18:14:52.432734, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6719> 2014-11-08 18:14:52.432812 7fc5adb11700 5 -- op tracker -- seq: 46968, time: 2014-11-08 18:14:52.432812, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(2a8cfcb5/1000002ad58.0000005a/head//4, version: 3468'6015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a8cfcb5/1000002ad58.0000005a/head//4@3468'6015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6718> 2014-11-08 18:14:52.450312 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3708 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:52.447528) v2 ==== 47+0+0 (1648109653 0 0) 0x227ad460 con 0x1c8b7d20 -6717> 2014-11-08 18:14:52.450339 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:52.447528) v2 -- ?+0 0x23144920 con 0x1c8b7d20 -6716> 2014-11-08 18:14:52.450398 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3708 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:52.447528) v2 ==== 47+0+0 (1648109653 0 0) 0x22e98000 con 0x1c9f8580 -6715> 2014-11-08 18:14:52.450410 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:52.447528) v2 -- ?+0 0x224a32a0 con 0x1c9f8580 -6714> 2014-11-08 18:14:52.490917 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239ed820 con 0x1c5c0580 -6713> 2014-11-08 18:14:52.677718 7fc5c362f700 5 osd.9 5773 tick -6712> 2014-11-08 18:14:52.816869 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3738 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:52.816370) v2 ==== 47+0+0 (2875338046 0 0) 0x23c34b00 con 0x1cce3180 -6711> 2014-11-08 18:14:52.816901 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:52.816370) v2 -- ?+0 0x22e98000 con 0x1cce3180 -6710> 2014-11-08 18:14:52.817054 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3738 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:52.816370) v2 ==== 47+0+0 (2875338046 0 0) 0x22d530c0 con 0x1cce3700 -6709> 2014-11-08 18:14:52.817069 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:52.816370) v2 -- ?+0 0x227ad460 con 0x1cce3700 -6708> 2014-11-08 18:14:52.994116 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 316 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:52.991732) v2 ==== 47+0+0 (76084149 0 0) 0x22997440 con 0x1eb9d6a0 -6707> 2014-11-08 18:14:52.994162 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:52.991732) v2 -- ?+0 0x23c34b00 con 0x1eb9d6a0 -6706> 2014-11-08 18:14:52.994338 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 316 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:52.991732) v2 ==== 47+0+0 (76084149 0 0) 0x1d5e4ec0 con 0x1eb9c780 -6705> 2014-11-08 18:14:52.994350 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:52.991732) v2 -- ?+0 0x22d530c0 con 0x1eb9c780 -6704> 2014-11-08 18:14:53.345642 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -6703> 2014-11-08 18:14:53.345701 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x4535fa0 con 0x1cae2940 -6702> 2014-11-08 18:14:53.345727 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x22908780 con 0x1cae2680 -6701> 2014-11-08 18:14:53.345736 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x257c4740 con 0x1c5c02c0 -6700> 2014-11-08 18:14:53.345747 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x227b2b20 con 0x1cae0160 -6699> 2014-11-08 18:14:53.345756 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1e894b00 con 0x1cae0f20 -6698> 2014-11-08 18:14:53.345766 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x257c30c0 con 0x1cae0c60 -6697> 2014-11-08 18:14:53.345775 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x2495e180 con 0x1cae23c0 -6696> 2014-11-08 18:14:53.345785 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x2583d0a0 con 0x1cae56a0 -6695> 2014-11-08 18:14:53.345894 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1f4c2b20 con 0x1cbf9ce0 -6694> 2014-11-08 18:14:53.345907 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1d4f2940 con 0x1d603020 -6693> 2014-11-08 18:14:53.345916 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1dbfddc0 con 0x1dbd9b80 -6692> 2014-11-08 18:14:53.345926 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1d326900 con 0x1d235d80 -6691> 2014-11-08 18:14:53.345937 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1deccec0 con 0x1cae1fa0 -6690> 2014-11-08 18:14:53.345946 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x25e83a20 con 0x1cae1ce0 -6689> 2014-11-08 18:14:53.345955 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x23d74380 con 0x1cae1340 -6688> 2014-11-08 18:14:53.345964 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x2290cce0 con 0x1cae1080 -6687> 2014-11-08 18:14:53.345973 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x257c41a0 con 0x1cae1a20 -6686> 2014-11-08 18:14:53.345983 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:53.345697) v2 -- ?+0 0x1db19fe0 con 0x1cae2c00 -6685> 2014-11-08 18:14:53.346409 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x258381e0 con 0x1cae56a0 -6684> 2014-11-08 18:14:53.346435 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x2495b660 con 0x1cae23c0 -6683> 2014-11-08 18:14:53.347086 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x229081e0 con 0x1cae0160 -6682> 2014-11-08 18:14:53.347102 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1d06dbe0 con 0x1cae1340 -6681> 2014-11-08 18:14:53.347112 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1d044ce0 con 0x1cae1080 -6680> 2014-11-08 18:14:53.347120 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1eace360 con 0x1cae1fa0 -6679> 2014-11-08 18:14:53.347129 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x4532d00 con 0x1cae1ce0 -6678> 2014-11-08 18:14:53.347138 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1d324ce0 con 0x1cae1a20 -6677> 2014-11-08 18:14:53.347149 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x2127dfa0 con 0x1cae2c00 -6676> 2014-11-08 18:14:53.347161 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x23681fe0 con 0x1cae2940 -6675> 2014-11-08 18:14:53.347170 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1d5e5640 con 0x1cae2680 -6674> 2014-11-08 18:14:53.347184 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 503 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x246c8f00 con 0x1cbf9ce0 -6673> 2014-11-08 18:14:53.347198 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 503 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x2234d460 con 0x1d603020 -6672> 2014-11-08 18:14:53.347210 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 310 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x2583d280 con 0x1dbd9b80 -6671> 2014-11-08 18:14:53.347222 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 310 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x2257a940 con 0x1d235d80 -6670> 2014-11-08 18:14:53.347232 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x257c3840 con 0x1c5c02c0 -6669> 2014-11-08 18:14:53.347243 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1e6aad00 con 0x1cae0f20 -6668> 2014-11-08 18:14:53.347253 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3723 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.345697) v2 ==== 47+0+0 (3466456266 0 0) 0x1e897620 con 0x1cae0c60 -6667> 2014-11-08 18:14:53.599031 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 496 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:53.596787) v2 ==== 47+0+0 (3580782977 0 0) 0x24883c00 con 0x1eb9c8e0 -6666> 2014-11-08 18:14:53.599088 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.596787) v2 -- ?+0 0x1d5e4ec0 con 0x1eb9c8e0 -6665> 2014-11-08 18:14:53.599299 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 496 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:53.596787) v2 ==== 47+0+0 (3580782977 0 0) 0x22e5c380 con 0x1eb9d800 -6664> 2014-11-08 18:14:53.599310 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.596787) v2 -- ?+0 0x22997440 con 0x1eb9d800 -6663> 2014-11-08 18:14:53.677833 7fc5c362f700 5 osd.9 5773 tick -6662> 2014-11-08 18:14:53.917518 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3739 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:53.916943) v2 ==== 47+0+0 (3823523355 0 0) 0x23c33840 con 0x1cce3180 -6661> 2014-11-08 18:14:53.917546 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.916943) v2 -- ?+0 0x22e5c380 con 0x1cce3180 -6660> 2014-11-08 18:14:53.918090 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3739 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:53.916943) v2 ==== 47+0+0 (3823523355 0 0) 0x22d50780 con 0x1cce3700 -6659> 2014-11-08 18:14:53.918103 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:53.916943) v2 -- ?+0 0x24883c00 con 0x1cce3700 -6658> 2014-11-08 18:14:54.377948 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(62586b16/1000001b629.0000000a/head//4),PushReplyOp(4a86b16/1000002af81.0000002d/head//4),PushReplyOp(e9a6b16/1000002a338.00000031/head//4),PushReplyOp(ada6b16/1000002b067.00000038/head//4),PushReplyOp(cacb6b16/1000002ac72.00000037/head//4)]) v2 -- ?+0 0x1ebb5c00 con 0x1c5c0580 -6657> 2014-11-08 18:14:54.379896 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(2a8cfcb5/1000002ad58.0000005a/head//4)]) v2 -- ?+0 0x1d3a8000 con 0x1c5c0580 -6656> 2014-11-08 18:14:54.380956 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8528 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) v3 ==== 757+0+0 (1400186571 0 0) 0x1d655580 con 0x1c5c0580 -6655> 2014-11-08 18:14:54.380971 7fc5a152e700 5 -- op tracker -- seq: 46969, time: 2014-11-08 18:14:54.380911, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6654> 2014-11-08 18:14:54.380980 7fc5a152e700 5 -- op tracker -- seq: 46969, time: 2014-11-08 18:14:54.380913, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6653> 2014-11-08 18:14:54.380985 7fc5a152e700 5 -- op tracker -- seq: 46969, time: 2014-11-08 18:14:54.380951, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6652> 2014-11-08 18:14:54.380990 7fc5a152e700 5 -- op tracker -- seq: 46969, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6651> 2014-11-08 18:14:54.381023 7fc5afb15700 5 -- op tracker -- seq: 46969, time: 2014-11-08 18:14:54.381023, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6650> 2014-11-08 18:14:54.381034 7fc5afb15700 5 -- op tracker -- seq: 46969, time: 2014-11-08 18:14:54.381034, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6649> 2014-11-08 18:14:54.381077 7fc5afb15700 5 -- op tracker -- seq: 46969, time: 2014-11-08 18:14:54.381077, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb cacb6b16/1000002ac72.00000037/head//4) -6648> 2014-11-08 18:14:54.383594 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8529 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) v3 ==== 757+0+0 (1232135471 0 0) 0x213f3180 con 0x1c5c0580 -6647> 2014-11-08 18:14:54.383609 7fc5a152e700 5 -- op tracker -- seq: 46970, time: 2014-11-08 18:14:54.383530, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6646> 2014-11-08 18:14:54.383615 7fc5a152e700 5 -- op tracker -- seq: 46970, time: 2014-11-08 18:14:54.383532, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6645> 2014-11-08 18:14:54.383620 7fc5a152e700 5 -- op tracker -- seq: 46970, time: 2014-11-08 18:14:54.383589, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6644> 2014-11-08 18:14:54.383625 7fc5a152e700 5 -- op tracker -- seq: 46970, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6643> 2014-11-08 18:14:54.383649 7fc5adb11700 5 -- op tracker -- seq: 46970, time: 2014-11-08 18:14:54.383649, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6642> 2014-11-08 18:14:54.383658 7fc5adb11700 5 -- op tracker -- seq: 46970, time: 2014-11-08 18:14:54.383658, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6641> 2014-11-08 18:14:54.383698 7fc5adb11700 5 -- op tracker -- seq: 46970, time: 2014-11-08 18:14:54.383698, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 2a8cfcb5/1000002ad58.0000005a/head//4) -6640> 2014-11-08 18:14:54.677951 7fc5c362f700 5 osd.9 5773 tick -6639> 2014-11-08 18:14:55.046428 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -6638> 2014-11-08 18:14:55.046467 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1dbf83c0 con 0x1cae2940 -6637> 2014-11-08 18:14:55.046494 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x246cb2a0 con 0x1cae2680 -6636> 2014-11-08 18:14:55.046506 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x2583c740 con 0x1c5c02c0 -6635> 2014-11-08 18:14:55.046516 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x2515d0a0 con 0x1cae0160 -6634> 2014-11-08 18:14:55.046526 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1f4c14a0 con 0x1cae0f20 -6633> 2014-11-08 18:14:55.046536 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1d4f03c0 con 0x1cae0c60 -6632> 2014-11-08 18:14:55.046545 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x4534ec0 con 0x1cae23c0 -6631> 2014-11-08 18:14:55.046555 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x25f96720 con 0x1cae56a0 -6630> 2014-11-08 18:14:55.046564 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1e897080 con 0x1cbf9ce0 -6629> 2014-11-08 18:14:55.046574 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x259f1a40 con 0x1d603020 -6628> 2014-11-08 18:14:55.046583 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1d8b9680 con 0x1dbd9b80 -6627> 2014-11-08 18:14:55.046592 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1e893840 con 0x1d235d80 -6626> 2014-11-08 18:14:55.046743 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1dbfeea0 con 0x1cae1fa0 -6625> 2014-11-08 18:14:55.046755 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x1eae3fc0 con 0x1cae1ce0 -6624> 2014-11-08 18:14:55.046764 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x2583c920 con 0x1cae1340 -6623> 2014-11-08 18:14:55.046773 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x2515c380 con 0x1cae1080 -6622> 2014-11-08 18:14:55.046782 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x2290fbc0 con 0x1cae1a20 -6621> 2014-11-08 18:14:55.046791 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:55.046463) v2 -- ?+0 0x257c4b00 con 0x1cae2c00 -6620> 2014-11-08 18:14:55.047237 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x2127ecc0 con 0x1cae2c00 -6619> 2014-11-08 18:14:55.047269 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x1d3223a0 con 0x1cae1a20 -6618> 2014-11-08 18:14:55.047282 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x259f12c0 con 0x1cae1340 -6617> 2014-11-08 18:14:55.047344 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x1d0432a0 con 0x1cae1080 -6616> 2014-11-08 18:14:55.048110 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x1eac9680 con 0x1cae1fa0 -6615> 2014-11-08 18:14:55.048159 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x4532b20 con 0x1cae1ce0 -6614> 2014-11-08 18:14:55.048174 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x1d5e4560 con 0x1cae2680 -6613> 2014-11-08 18:14:55.048398 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x23687260 con 0x1cae2940 -6612> 2014-11-08 18:14:55.048411 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x257c6540 con 0x1c5c02c0 -6611> 2014-11-08 18:14:55.048432 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x22909c20 con 0x1cae0160 -6610> 2014-11-08 18:14:55.048642 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x1e6ab2a0 con 0x1cae0f20 -6609> 2014-11-08 18:14:55.048656 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x1e8901e0 con 0x1cae0c60 -6608> 2014-11-08 18:14:55.049103 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 504 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x2234eae0 con 0x1d603020 -6607> 2014-11-08 18:14:55.049138 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x2583d820 con 0x1cae56a0 -6606> 2014-11-08 18:14:55.049156 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3724 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x2495bc00 con 0x1cae23c0 -6605> 2014-11-08 18:14:55.049183 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 504 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x246ce540 con 0x1cbf9ce0 -6604> 2014-11-08 18:14:55.049431 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 311 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x2257eea0 con 0x1d235d80 -6603> 2014-11-08 18:14:55.049445 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 311 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:55.046463) v2 ==== 47+0+0 (2795081701 0 0) 0x25838f00 con 0x1dbd9b80 -6602> 2014-11-08 18:14:55.202285 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8530 ==== MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (2466250367 0 0) 0x1d3a8000 con 0x1c5c0580 -6601> 2014-11-08 18:14:55.202333 7fc5a152e700 5 -- op tracker -- seq: 46971, time: 2014-11-08 18:14:55.080086, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6600> 2014-11-08 18:14:55.202352 7fc5a152e700 5 -- op tracker -- seq: 46971, time: 2014-11-08 18:14:55.080091, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6599> 2014-11-08 18:14:55.202366 7fc5a152e700 5 -- op tracker -- seq: 46971, time: 2014-11-08 18:14:55.202273, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6598> 2014-11-08 18:14:55.202380 7fc5a152e700 5 -- op tracker -- seq: 46971, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6597> 2014-11-08 18:14:55.202438 7fc5ad310700 5 -- op tracker -- seq: 46971, time: 2014-11-08 18:14:55.202438, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6596> 2014-11-08 18:14:55.202566 7fc5ad310700 5 -- op tracker -- seq: 46971, time: 2014-11-08 18:14:55.202566, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(facb6b16/1000001aaae.0000004d/head//4, version: 2829'2415, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(facb6b16/1000001aaae.0000004d/head//4@2829'2415, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2db6b16/1000002b690.000004cd/head//4, version: 3473'6835, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2db6b16/1000002b690.000004cd/head//4@3473'6835, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b08d6b16/1000002bc88.00000079/head//4, version: 3485'7644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b08d6b16/1000002bc88.00000079/head//4@3485'7644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6595> 2014-11-08 18:14:55.352012 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239e83c0 con 0x1c5c0580 -6594> 2014-11-08 18:14:55.352049 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239eb480 con 0x1c5c0580 -6593> 2014-11-08 18:14:55.352058 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ea1c0 con 0x1c5c0580 -6592> 2014-11-08 18:14:55.352069 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(facb6b16/1000001aaae.0000004d/head//4),PushReplyOp(2db6b16/1000002b690.000004cd/head//4),PushReplyOp(b08d6b16/1000002bc88.00000079/head//4)]) v2 -- ?+0 0x229bba00 con 0x1c5c0580 -6591> 2014-11-08 18:14:55.357800 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8531 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) v3 ==== 757+0+0 (3509043964 0 0) 0x213f6c00 con 0x1c5c0580 -6590> 2014-11-08 18:14:55.357821 7fc5a152e700 5 -- op tracker -- seq: 46972, time: 2014-11-08 18:14:55.357729, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6589> 2014-11-08 18:14:55.357829 7fc5a152e700 5 -- op tracker -- seq: 46972, time: 2014-11-08 18:14:55.357732, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6588> 2014-11-08 18:14:55.357832 7fc5a152e700 5 -- op tracker -- seq: 46972, time: 2014-11-08 18:14:55.357795, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6587> 2014-11-08 18:14:55.357835 7fc5a152e700 5 -- op tracker -- seq: 46972, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6586> 2014-11-08 18:14:55.357876 7fc5afb15700 5 -- op tracker -- seq: 46972, time: 2014-11-08 18:14:55.357876, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6585> 2014-11-08 18:14:55.357888 7fc5afb15700 5 -- op tracker -- seq: 46972, time: 2014-11-08 18:14:55.357888, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6584> 2014-11-08 18:14:55.357937 7fc5afb15700 5 -- op tracker -- seq: 46972, time: 2014-11-08 18:14:55.357937, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb b08d6b16/1000002bc88.00000079/head//4) -6583> 2014-11-08 18:14:55.678119 7fc5c362f700 5 osd.9 5773 tick -6582> 2014-11-08 18:14:56.166720 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2214 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) v2 ==== 110+0+0 (1799118064 0 0) 0x24840c00 con 0x1e50aec0 -6581> 2014-11-08 18:14:56.166755 7fc5a56ef700 5 -- op tracker -- seq: 46973, time: 2014-11-08 18:14:56.166608, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) -6580> 2014-11-08 18:14:56.166764 7fc5a56ef700 5 -- op tracker -- seq: 46973, time: 2014-11-08 18:14:56.166612, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) -6579> 2014-11-08 18:14:56.166768 7fc5a56ef700 5 -- op tracker -- seq: 46973, time: 2014-11-08 18:14:56.166714, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) -6578> 2014-11-08 18:14:56.166772 7fc5a56ef700 5 -- op tracker -- seq: 46973, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) -6577> 2014-11-08 18:14:56.166808 7fc5adb11700 5 -- op tracker -- seq: 46973, time: 2014-11-08 18:14:56.166808, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) -6576> 2014-11-08 18:14:56.166887 7fc5adb11700 5 -- op tracker -- seq: 46973, time: 2014-11-08 18:14:56.166887, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(8c4e1d83/1000001f006.00000000/head//4)]) -6575> 2014-11-08 18:14:56.172651 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2215 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) v2 ==== 305+0+0 (2693548157 0 0) 0x1d5b2200 con 0x1e50aec0 -6574> 2014-11-08 18:14:56.172666 7fc5a56ef700 5 -- op tracker -- seq: 46974, time: 2014-11-08 18:14:56.172590, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -6573> 2014-11-08 18:14:56.172674 7fc5a56ef700 5 -- op tracker -- seq: 46974, time: 2014-11-08 18:14:56.172591, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -6572> 2014-11-08 18:14:56.172680 7fc5a56ef700 5 -- op tracker -- seq: 46974, time: 2014-11-08 18:14:56.172648, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -6571> 2014-11-08 18:14:56.172692 7fc5a56ef700 5 -- op tracker -- seq: 46974, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -6570> 2014-11-08 18:14:56.172716 7fc5ac30e700 5 -- op tracker -- seq: 46974, time: 2014-11-08 18:14:56.172716, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -6569> 2014-11-08 18:14:56.172823 7fc5ac30e700 5 -- op tracker -- seq: 46974, time: 2014-11-08 18:14:56.172823, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(a658dd13/1000001a207.000005b8/head//4),PushReplyOp(fd19dd13/1000002a5c7.00000003/head//4),PushReplyOp(da99dd13/1000000e4b7.000003ed/head//4),PushReplyOp(88aadd13/10000029db8.0000000b/head//4)]) -6568> 2014-11-08 18:14:56.404419 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3676 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:56.401725) v2 ==== 47+0+0 (2649854704 0 0) 0x224a6720 con 0x1cce2ec0 -6567> 2014-11-08 18:14:56.404450 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3676 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:56.401725) v2 ==== 47+0+0 (2649854704 0 0) 0x23147bc0 con 0x1cce3440 -6566> 2014-11-08 18:14:56.404465 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.401725) v2 -- ?+0 0x23c33840 con 0x1cce2ec0 -6565> 2014-11-08 18:14:56.404536 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.401725) v2 -- ?+0 0x22d50780 con 0x1cce3440 -6564> 2014-11-08 18:14:56.678235 7fc5c362f700 5 osd.9 5773 tick -6563> 2014-11-08 18:14:56.747175 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -6562> 2014-11-08 18:14:56.747197 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1db1bde0 con 0x1cae2940 -6561> 2014-11-08 18:14:56.747216 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1dbf8780 con 0x1cae2680 -6560> 2014-11-08 18:14:56.747227 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1d322760 con 0x1c5c02c0 -6559> 2014-11-08 18:14:56.747237 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x4537260 con 0x1cae0160 -6558> 2014-11-08 18:14:56.747248 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x25fd2b20 con 0x1cae0f20 -6557> 2014-11-08 18:14:56.747258 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1decc920 con 0x1cae0c60 -6556> 2014-11-08 18:14:56.747268 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x22456900 con 0x1cae23c0 -6555> 2014-11-08 18:14:56.747278 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1d326720 con 0x1cae56a0 -6554> 2014-11-08 18:14:56.747287 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1f4c2ee0 con 0x1cbf9ce0 -6553> 2014-11-08 18:14:56.747298 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1d4f01e0 con 0x1d603020 -6552> 2014-11-08 18:14:56.747307 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x22452ee0 con 0x1dbd9b80 -6551> 2014-11-08 18:14:56.747316 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1dbfe720 con 0x1d235d80 -6550> 2014-11-08 18:14:56.747325 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x2234e720 con 0x1cae1fa0 -6549> 2014-11-08 18:14:56.747333 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1dec9680 con 0x1cae1ce0 -6548> 2014-11-08 18:14:56.747342 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x24886720 con 0x1cae1340 -6547> 2014-11-08 18:14:56.747350 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1dbfd820 con 0x1cae1080 -6546> 2014-11-08 18:14:56.747359 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x1eae3a20 con 0x1cae1a20 -6545> 2014-11-08 18:14:56.747368 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:14:56.747194) v2 -- ?+0 0x2515eae0 con 0x1cae2c00 -6544> 2014-11-08 18:14:56.747960 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x236850a0 con 0x1cae2940 -6543> 2014-11-08 18:14:56.747981 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 505 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x223494a0 con 0x1d603020 -6542> 2014-11-08 18:14:56.748071 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x229094a0 con 0x1cae0160 -6541> 2014-11-08 18:14:56.748083 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x1e6a81e0 con 0x1cae0f20 -6540> 2014-11-08 18:14:56.748092 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x1e8923a0 con 0x1cae0c60 -6539> 2014-11-08 18:14:56.748101 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x259f1e00 con 0x1cae1340 -6538> 2014-11-08 18:14:56.748110 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x1eacb660 con 0x1cae1fa0 -6537> 2014-11-08 18:14:56.748119 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x4533840 con 0x1cae1ce0 -6536> 2014-11-08 18:14:56.748127 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x2495bde0 con 0x1cae23c0 -6535> 2014-11-08 18:14:56.748136 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x2583a940 con 0x1cae56a0 -6534> 2014-11-08 18:14:56.748144 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 505 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x246c83c0 con 0x1cbf9ce0 -6533> 2014-11-08 18:14:56.748203 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x257c4380 con 0x1c5c02c0 -6532> 2014-11-08 18:14:56.748215 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x1d046ea0 con 0x1cae1080 -6531> 2014-11-08 18:14:56.748351 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x1d5e5be0 con 0x1cae2680 -6530> 2014-11-08 18:14:56.748365 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 312 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x2583dbe0 con 0x1dbd9b80 -6529> 2014-11-08 18:14:56.748375 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 312 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x2026f440 con 0x1d235d80 -6528> 2014-11-08 18:14:56.748384 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x2127fda0 con 0x1cae2c00 -6527> 2014-11-08 18:14:56.748438 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3725 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:56.747194) v2 ==== 47+0+0 (2124863927 0 0) 0x1d325280 con 0x1cae1a20 -6526> 2014-11-08 18:14:57.094255 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 317 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:57.092506) v2 ==== 47+0+0 (1226086152 0 0) 0x1d5e0780 con 0x1eb9c780 -6525> 2014-11-08 18:14:57.094302 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:57.092506) v2 -- ?+0 0x23147bc0 con 0x1eb9c780 -6524> 2014-11-08 18:14:57.095406 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 317 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:57.092506) v2 ==== 47+0+0 (1226086152 0 0) 0x212130c0 con 0x1eb9d6a0 -6523> 2014-11-08 18:14:57.095419 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:57.092506) v2 -- ?+0 0x224a6720 con 0x1eb9d6a0 -6522> 2014-11-08 18:14:57.100490 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 497 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:57.098390) v2 ==== 47+0+0 (1972219956 0 0) 0x1e6ae180 con 0x1eb9d800 -6521> 2014-11-08 18:14:57.100503 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:57.098390) v2 -- ?+0 0x212130c0 con 0x1eb9d800 -6520> 2014-11-08 18:14:57.100513 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 497 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:57.098390) v2 ==== 47+0+0 (1972219956 0 0) 0x1e6afda0 con 0x1eb9c8e0 -6519> 2014-11-08 18:14:57.100651 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:57.098390) v2 -- ?+0 0x1d5e0780 con 0x1eb9c8e0 -6518> 2014-11-08 18:14:57.150783 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3709 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:57.148310) v2 ==== 47+0+0 (2411770285 0 0) 0x227aecc0 con 0x1c8b7d20 -6517> 2014-11-08 18:14:57.150799 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:57.148310) v2 -- ?+0 0x1e6afda0 con 0x1c8b7d20 -6516> 2014-11-08 18:14:57.150897 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3709 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:57.148310) v2 ==== 47+0+0 (2411770285 0 0) 0x1ec21680 con 0x1c9f8580 -6515> 2014-11-08 18:14:57.150909 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:57.148310) v2 -- ?+0 0x1e6ae180 con 0x1c9f8580 -6514> 2014-11-08 18:14:57.507556 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=cd9e1d83/1000001a9f6.00000159/head//4)) v1 -- ?+0 0x2302f740 con 0x1c5c0580 -6513> 2014-11-08 18:14:57.507599 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x23029200 con 0x42c86e0 -6512> 2014-11-08 18:14:57.507615 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x23029200 con 0x42c86e0 -6511> 2014-11-08 18:14:57.507612 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 75be1d83/100000265df.00000001/head//4 -6510> 2014-11-08 18:14:57.507631 7fc5b9328700 5 -- op tracker -- seq: 46975, time: 2014-11-08 18:14:57.507607, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) -6509> 2014-11-08 18:14:57.507641 7fc5b9328700 5 -- op tracker -- seq: 46975, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) -6508> 2014-11-08 18:14:57.507643 7fc5b9328700 5 -- op tracker -- seq: 46975, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) -6507> 2014-11-08 18:14:57.507646 7fc5b9328700 5 -- op tracker -- seq: 46975, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) -6506> 2014-11-08 18:14:57.507640 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb 8c4e1d83/1000001f006.00000000/head//4) v3 -- ?+0 0x1e58da00 con 0x1e50aec0 -6505> 2014-11-08 18:14:57.507867 7fc5b0316700 5 -- op tracker -- seq: 46975, time: 2014-11-08 18:14:57.507867, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) -6504> 2014-11-08 18:14:57.540831 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8532 ==== MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (2725540328 0 0) 0x229bba00 con 0x1c5c0580 -6503> 2014-11-08 18:14:57.540880 7fc5a152e700 5 -- op tracker -- seq: 46976, time: 2014-11-08 18:14:57.435195, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6502> 2014-11-08 18:14:57.540900 7fc5a152e700 5 -- op tracker -- seq: 46976, time: 2014-11-08 18:14:57.435201, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6501> 2014-11-08 18:14:57.540913 7fc5a152e700 5 -- op tracker -- seq: 46976, time: 2014-11-08 18:14:57.540820, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6500> 2014-11-08 18:14:57.540926 7fc5a152e700 5 -- op tracker -- seq: 46976, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6499> 2014-11-08 18:14:57.540971 7fc5ad310700 5 -- op tracker -- seq: 46976, time: 2014-11-08 18:14:57.540971, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6498> 2014-11-08 18:14:57.541100 7fc5ad310700 5 -- op tracker -- seq: 46976, time: 2014-11-08 18:14:57.541099, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(7ebd6b16/1000001a670.0000005a/head//4, version: 397'2057, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ebd6b16/1000001a670.0000005a/head//4@397'2057, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(693e6b16/10000029533.0000009a/head//4, version: 3033'4098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(693e6b16/10000029533.0000009a/head//4@3033'4098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f66e6b16/10000019f97.00000469/head//4, version: 397'1504, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f66e6b16/10000019f97.00000469/head//4@397'1504, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6497> 2014-11-08 18:14:57.609659 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2216 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) v2 ==== 370+0+0 (4255715585 0 0) 0x229b9200 con 0x1e50aec0 -6496> 2014-11-08 18:14:57.609709 7fc5a56ef700 5 -- op tracker -- seq: 46977, time: 2014-11-08 18:14:57.609576, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) -6495> 2014-11-08 18:14:57.609721 7fc5a56ef700 5 -- op tracker -- seq: 46977, time: 2014-11-08 18:14:57.609580, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) -6494> 2014-11-08 18:14:57.609728 7fc5a56ef700 5 -- op tracker -- seq: 46977, time: 2014-11-08 18:14:57.609651, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) -6493> 2014-11-08 18:14:57.609733 7fc5a56ef700 5 -- op tracker -- seq: 46977, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) -6492> 2014-11-08 18:14:57.609794 7fc5afb15700 5 -- op tracker -- seq: 46977, time: 2014-11-08 18:14:57.609794, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) -6491> 2014-11-08 18:14:57.609913 7fc5afb15700 5 -- op tracker -- seq: 46977, time: 2014-11-08 18:14:57.609913, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(fa3ed484/1000002a7de.00000070/head//4),PushReplyOp(1f4fd484/1000002b091.00000098/head//4),PushReplyOp(2b7fd484/1000001a64d.00000095/head//4),PushReplyOp(ddafd484/1000001950e.00000013/head//4),PushReplyOp(f9bfd484/1000002a825.000000bd/head//4)]) -6490> 2014-11-08 18:14:57.678393 7fc5c362f700 5 osd.9 5773 tick -6489> 2014-11-08 18:14:57.690993 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2217 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) v2 ==== 240+0+0 (1280925099 0 0) 0x1d85c200 con 0x1e50aec0 -6488> 2014-11-08 18:14:57.691020 7fc5a56ef700 5 -- op tracker -- seq: 46978, time: 2014-11-08 18:14:57.690900, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) -6487> 2014-11-08 18:14:57.691031 7fc5a56ef700 5 -- op tracker -- seq: 46978, time: 2014-11-08 18:14:57.690904, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) -6486> 2014-11-08 18:14:57.691037 7fc5a56ef700 5 -- op tracker -- seq: 46978, time: 2014-11-08 18:14:57.690985, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) -6485> 2014-11-08 18:14:57.691042 7fc5a56ef700 5 -- op tracker -- seq: 46978, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) -6484> 2014-11-08 18:14:57.691099 7fc5ac30e700 5 -- op tracker -- seq: 46978, time: 2014-11-08 18:14:57.691099, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) -6483> 2014-11-08 18:14:57.691302 7fc5ac30e700 5 -- op tracker -- seq: 46978, time: 2014-11-08 18:14:57.691302, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(2ddff418/1000002845d.00000061/head//4),PushReplyOp(26810518/1000001a239.00000407/head//4),PushReplyOp(20020518/1000001aac2.0000001a/head//4)]) -6482> 2014-11-08 18:14:57.697786 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) v1 -- ?+0 0x224beb80 con 0x42c86e0 -6481> 2014-11-08 18:14:57.697821 7fc5b0316700 5 -- op tracker -- seq: 46975, time: 2014-11-08 18:14:57.697821, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4409, to_read={cd9e1d83/1000001a9f6.00000159/head//4=0,4194304}, attrs_to_read=)) -6480> 2014-11-08 18:14:57.697823 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224beb80 con 0x42c86e0 -6479> 2014-11-08 18:14:57.697834 7fc5b9328700 5 -- op tracker -- seq: 46979, time: 2014-11-08 18:14:57.697814, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) -6478> 2014-11-08 18:14:57.697838 7fc5b9328700 5 -- op tracker -- seq: 46979, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) -6477> 2014-11-08 18:14:57.697840 7fc5b9328700 5 -- op tracker -- seq: 46979, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) -6476> 2014-11-08 18:14:57.697842 7fc5b9328700 5 -- op tracker -- seq: 46979, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) -6475> 2014-11-08 18:14:57.697856 7fc5b0316700 5 -- op tracker -- seq: 46979, time: 2014-11-08 18:14:57.697856, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) -6474> 2014-11-08 18:14:57.697880 7fc5b0316700 5 -- op tracker -- seq: 46979, time: 2014-11-08 18:14:57.697880, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=0)) -6473> 2014-11-08 18:14:57.726263 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=e71bdd13/1000001a918.00000034/head//4,df3cdd13/10000029ac9.0000000c/head//4,fa7ddd13/1000002a611.00000042/head//4,65addd13/10000019aef.00000433/head//4)) v1 -- ?+0 0x2302c5c0 con 0x1e50aec0 -6472> 2014-11-08 18:14:57.726302 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2302cc80 con 0x42c86e0 -6471> 2014-11-08 18:14:57.726318 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=4 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 730edd13/1000002ab43.00000046/head//4 -6470> 2014-11-08 18:14:57.726345 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb 88aadd13/10000029db8.0000000b/head//4) v3 -- ?+0 0x1ce9de80 con 0x1c5c0580 -6469> 2014-11-08 18:14:57.726357 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb 88aadd13/10000029db8.0000000b/head//4) v3 -- ?+0 0x1ce98480 con 0x1e50aec0 -6468> 2014-11-08 18:14:57.726369 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2302cc80 con 0x42c86e0 -6467> 2014-11-08 18:14:57.726386 7fc5b9328700 5 -- op tracker -- seq: 46980, time: 2014-11-08 18:14:57.726311, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) -6466> 2014-11-08 18:14:57.726396 7fc5b9328700 5 -- op tracker -- seq: 46980, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) -6465> 2014-11-08 18:14:57.726400 7fc5b9328700 5 -- op tracker -- seq: 46980, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) -6464> 2014-11-08 18:14:57.726404 7fc5b9328700 5 -- op tracker -- seq: 46980, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) -6463> 2014-11-08 18:14:57.726438 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239e94a0 con 0x1c5c0580 -6462> 2014-11-08 18:14:57.726447 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ed0a0 con 0x1c5c0580 -6461> 2014-11-08 18:14:57.726453 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ef080 con 0x1c5c0580 -6460> 2014-11-08 18:14:57.726458 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(7ebd6b16/1000001a670.0000005a/head//4),PushReplyOp(693e6b16/10000029533.0000009a/head//4),PushReplyOp(f66e6b16/10000019f97.00000469/head//4)]) v2 -- ?+0 0x1d3a8000 con 0x1c5c0580 -6459> 2014-11-08 18:14:57.726697 7fc5aeb13700 5 -- op tracker -- seq: 46980, time: 2014-11-08 18:14:57.726697, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) -6458> 2014-11-08 18:14:57.731769 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8533 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) v3 ==== 757+0+0 (3660159423 0 0) 0x1ce9de80 con 0x1c5c0580 -6457> 2014-11-08 18:14:57.731785 7fc5a152e700 5 -- op tracker -- seq: 46981, time: 2014-11-08 18:14:57.731726, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6456> 2014-11-08 18:14:57.731791 7fc5a152e700 5 -- op tracker -- seq: 46981, time: 2014-11-08 18:14:57.731728, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6455> 2014-11-08 18:14:57.731795 7fc5a152e700 5 -- op tracker -- seq: 46981, time: 2014-11-08 18:14:57.731765, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6454> 2014-11-08 18:14:57.731799 7fc5a152e700 5 -- op tracker -- seq: 46981, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6453> 2014-11-08 18:14:57.731821 7fc5ad310700 5 -- op tracker -- seq: 46981, time: 2014-11-08 18:14:57.731821, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6452> 2014-11-08 18:14:57.731831 7fc5ad310700 5 -- op tracker -- seq: 46981, time: 2014-11-08 18:14:57.731831, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6451> 2014-11-08 18:14:57.731867 7fc5ad310700 5 -- op tracker -- seq: 46981, time: 2014-11-08 18:14:57.731867, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f66e6b16/10000019f97.00000469/head//4) -6450> 2014-11-08 18:14:57.815193 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2218 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) v3 ==== 757+0+0 (993561197 0 0) 0x1ce98480 con 0x1e50aec0 -6449> 2014-11-08 18:14:57.815214 7fc5a56ef700 5 -- op tracker -- seq: 46982, time: 2014-11-08 18:14:57.815114, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6448> 2014-11-08 18:14:57.815222 7fc5a56ef700 5 -- op tracker -- seq: 46982, time: 2014-11-08 18:14:57.815118, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6447> 2014-11-08 18:14:57.815225 7fc5a56ef700 5 -- op tracker -- seq: 46982, time: 2014-11-08 18:14:57.815188, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6446> 2014-11-08 18:14:57.815229 7fc5a56ef700 5 -- op tracker -- seq: 46982, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6445> 2014-11-08 18:14:57.815257 7fc5acb0f700 5 -- op tracker -- seq: 46982, time: 2014-11-08 18:14:57.815257, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6444> 2014-11-08 18:14:57.815269 7fc5acb0f700 5 -- op tracker -- seq: 46982, time: 2014-11-08 18:14:57.815269, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6443> 2014-11-08 18:14:57.815311 7fc5acb0f700 5 -- op tracker -- seq: 46982, time: 2014-11-08 18:14:57.815311, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 42eb90b7/1000002a8e5.0000005c/head//4) -6442> 2014-11-08 18:14:57.859847 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=8c820518/10000019a25.000003f3/head//4,29040518/1000002a743.00000084/head//4,f6240518/1000002b12f.00000140/head//4,37440518/1000002a55c.00000089/head//4,da050518/1000002a9cd.00000050/head//4)) v1 -- ?+0 0x23028b40 con 0x1c5c0580 -6441> 2014-11-08 18:14:57.859888 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2302f2c0 con 0x42c86e0 -6440> 2014-11-08 18:14:57.859904 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 44260518/1000002ad5b.00000044/head//4 -6439> 2014-11-08 18:14:57.859907 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2302f2c0 con 0x42c86e0 -6438> 2014-11-08 18:14:57.859929 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb 20020518/1000001aac2.0000001a/head//4) v3 -- ?+0 0x1ce9d100 con 0x1e50aec0 -6437> 2014-11-08 18:14:57.859927 7fc5b9328700 5 -- op tracker -- seq: 46983, time: 2014-11-08 18:14:57.859896, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) -6436> 2014-11-08 18:14:57.859938 7fc5b9328700 5 -- op tracker -- seq: 46983, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) -6435> 2014-11-08 18:14:57.859942 7fc5b9328700 5 -- op tracker -- seq: 46983, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) -6434> 2014-11-08 18:14:57.859947 7fc5b9328700 5 -- op tracker -- seq: 46983, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) -6433> 2014-11-08 18:14:57.860104 7fc5ac30e700 5 -- op tracker -- seq: 46983, time: 2014-11-08 18:14:57.860104, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) -6432> 2014-11-08 18:14:58.018079 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3740 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:58.017704) v2 ==== 47+0+0 (4263128149 0 0) 0x23c31c20 con 0x1cce3180 -6431> 2014-11-08 18:14:58.018114 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:58.017704) v2 -- ?+0 0x1ec21680 con 0x1cce3180 -6430> 2014-11-08 18:14:58.018837 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3740 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:58.017704) v2 ==== 47+0+0 (4263128149 0 0) 0x2485e180 con 0x1cce3700 -6429> 2014-11-08 18:14:58.018852 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:58.017704) v2 -- ?+0 0x227aecc0 con 0x1cce3700 -6428> 2014-11-08 18:14:58.061402 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8534 ==== MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1946204493 0 0) 0x1d3a8000 con 0x1c5c0580 -6427> 2014-11-08 18:14:58.061436 7fc5a152e700 5 -- op tracker -- seq: 46984, time: 2014-11-08 18:14:58.015240, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6426> 2014-11-08 18:14:58.061449 7fc5a152e700 5 -- op tracker -- seq: 46984, time: 2014-11-08 18:14:58.015244, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6425> 2014-11-08 18:14:58.061457 7fc5a152e700 5 -- op tracker -- seq: 46984, time: 2014-11-08 18:14:58.061379, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6424> 2014-11-08 18:14:58.061465 7fc5a152e700 5 -- op tracker -- seq: 46984, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6423> 2014-11-08 18:14:58.061502 7fc5afb15700 5 -- op tracker -- seq: 46984, time: 2014-11-08 18:14:58.061502, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6422> 2014-11-08 18:14:58.061582 7fc5afb15700 5 -- op tracker -- seq: 46984, time: 2014-11-08 18:14:58.061582, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(32ae6b16/10000029861.0000002d/head//4, version: 3033'4542, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(32ae6b16/10000029861.0000002d/head//4@3033'4542, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6421> 2014-11-08 18:14:58.115283 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8535 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) v1 ==== 2097696+0+0 (2905094424 0 0) 0x21a4c100 con 0x1c5c0580 -6420> 2014-11-08 18:14:58.115316 7fc5a152e700 5 -- op tracker -- seq: 46985, time: 2014-11-08 18:14:58.066657, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) -6419> 2014-11-08 18:14:58.115325 7fc5a152e700 5 -- op tracker -- seq: 46985, time: 2014-11-08 18:14:58.066660, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) -6418> 2014-11-08 18:14:58.115328 7fc5a152e700 5 -- op tracker -- seq: 46985, time: 2014-11-08 18:14:58.115274, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) -6417> 2014-11-08 18:14:58.115330 7fc5a152e700 5 -- op tracker -- seq: 46985, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) -6416> 2014-11-08 18:14:58.115532 7fc5adb11700 5 -- op tracker -- seq: 46985, time: 2014-11-08 18:14:58.115532, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) -6415> 2014-11-08 18:14:58.121059 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(cd9e1d83/1000001a9f6.00000159/head//4, version: 2829'2347, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cd9e1d83/1000001a9f6.00000159/head//4@2829'2347, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393aa00 con 0x1e50aec0 -6414> 2014-11-08 18:14:58.124776 7fc5adb11700 5 -- op tracker -- seq: 46985, time: 2014-11-08 18:14:58.124775, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4409, attrs_read=1)) -6413> 2014-11-08 18:14:58.125683 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=6dfd484/1000001abf1.00000009/head//4,1cefd484/1000002bd64.00000049/head//4,b2a0e484/1000002bd57.00000060/head//4)) v1 -- ?+0 0x1ea78d80 con 0x1c5c0580 -6412> 2014-11-08 18:14:58.125884 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7f500 con 0x42c86e0 -6411> 2014-11-08 18:14:58.125899 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 1bd0e484/1000002aaa9.00000053/head//4 -6410> 2014-11-08 18:14:58.125925 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb f9bfd484/1000002a825.000000bd/head//4) v3 -- ?+0 0x1e58b180 con 0x1e50aec0 -6409> 2014-11-08 18:14:58.125949 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7f500 con 0x42c86e0 -6408> 2014-11-08 18:14:58.125961 7fc5b9328700 5 -- op tracker -- seq: 46986, time: 2014-11-08 18:14:58.125892, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) -6407> 2014-11-08 18:14:58.125967 7fc5b9328700 5 -- op tracker -- seq: 46986, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) -6406> 2014-11-08 18:14:58.125971 7fc5b9328700 5 -- op tracker -- seq: 46986, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) -6405> 2014-11-08 18:14:58.125975 7fc5b9328700 5 -- op tracker -- seq: 46986, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) -6404> 2014-11-08 18:14:58.126009 7fc5ad310700 5 -- op tracker -- seq: 46986, time: 2014-11-08 18:14:58.126009, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) -6403> 2014-11-08 18:14:58.235703 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) v1 -- ?+0 0x44a6e00 con 0x42c86e0 -6402> 2014-11-08 18:14:58.235759 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x44a6e00 con 0x42c86e0 -6401> 2014-11-08 18:14:58.235758 7fc5aeb13700 5 -- op tracker -- seq: 46980, time: 2014-11-08 18:14:58.235758, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4410, to_read={e71bdd13/1000001a918.00000034/head//4=0,4194304,df3cdd13/10000029ac9.0000000c/head//4=0,4194304,fa7ddd13/1000002a611.00000042/head//4=0,4194304,65addd13/10000019aef.00000433/head//4=0,4194304}, attrs_to_read=)) -6400> 2014-11-08 18:14:58.235779 7fc5b9328700 5 -- op tracker -- seq: 46987, time: 2014-11-08 18:14:58.235744, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) -6399> 2014-11-08 18:14:58.235783 7fc5b9328700 5 -- op tracker -- seq: 46987, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) -6398> 2014-11-08 18:14:58.235785 7fc5b9328700 5 -- op tracker -- seq: 46987, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) -6397> 2014-11-08 18:14:58.235787 7fc5b9328700 5 -- op tracker -- seq: 46987, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) -6396> 2014-11-08 18:14:58.235818 7fc5aeb13700 5 -- op tracker -- seq: 46987, time: 2014-11-08 18:14:58.235818, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) -6395> 2014-11-08 18:14:58.235888 7fc5aeb13700 5 -- op tracker -- seq: 46987, time: 2014-11-08 18:14:58.235887, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=0)) -6394> 2014-11-08 18:14:58.473264 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) v1 -- ?+0 0x211d0f00 con 0x42c86e0 -6393> 2014-11-08 18:14:58.473308 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x211d0f00 con 0x42c86e0 -6392> 2014-11-08 18:14:58.473322 7fc5b9328700 5 -- op tracker -- seq: 46988, time: 2014-11-08 18:14:58.473296, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) -6391> 2014-11-08 18:14:58.473332 7fc5b9328700 5 -- op tracker -- seq: 46988, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) -6390> 2014-11-08 18:14:58.473336 7fc5b9328700 5 -- op tracker -- seq: 46988, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) -6389> 2014-11-08 18:14:58.473338 7fc5b9328700 5 -- op tracker -- seq: 46988, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) -6388> 2014-11-08 18:14:58.473379 7fc5afb15700 5 -- op tracker -- seq: 46988, time: 2014-11-08 18:14:58.473379, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) -6387> 2014-11-08 18:14:58.473499 7fc5afb15700 5 -- op tracker -- seq: 46988, time: 2014-11-08 18:14:58.473499, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=0)) -6386> 2014-11-08 18:14:58.473598 7fc5ad310700 5 -- op tracker -- seq: 46986, time: 2014-11-08 18:14:58.473598, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4412, to_read={6dfd484/1000001abf1.00000009/head//4=0,4194304,1cefd484/1000002bd64.00000049/head//4=0,4194304,b2a0e484/1000002bd57.00000060/head//4=0,4194304}, attrs_to_read=)) -6385> 2014-11-08 18:14:58.475861 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12060 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 377+0+0 (2941219611 0 0) 0x224ab180 con 0x1c5c2680 -6384> 2014-11-08 18:14:58.475895 7fc5a6cfe700 5 -- op tracker -- seq: 46989, time: 2014-11-08 18:14:58.475763, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) -6383> 2014-11-08 18:14:58.475905 7fc5a6cfe700 5 -- op tracker -- seq: 46989, time: 2014-11-08 18:14:58.475767, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) -6382> 2014-11-08 18:14:58.475910 7fc5a6cfe700 5 -- op tracker -- seq: 46989, time: 2014-11-08 18:14:58.475853, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) -6381> 2014-11-08 18:14:58.475915 7fc5a6cfe700 5 -- op tracker -- seq: 46989, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) -6380> 2014-11-08 18:14:58.477961 7fc5aeb13700 5 -- op tracker -- seq: 46989, time: 2014-11-08 18:14:58.477961, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) -6379> 2014-11-08 18:14:58.484600 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2219 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) v1 ==== 8390589+0+0 (617883309 0 0) 0x44a1b80 con 0x1e50aec0 -6378> 2014-11-08 18:14:58.484655 7fc5a56ef700 5 -- op tracker -- seq: 46990, time: 2014-11-08 18:14:58.365177, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) -6377> 2014-11-08 18:14:58.484669 7fc5a56ef700 5 -- op tracker -- seq: 46990, time: 2014-11-08 18:14:58.365182, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) -6376> 2014-11-08 18:14:58.484679 7fc5a56ef700 5 -- op tracker -- seq: 46990, time: 2014-11-08 18:14:58.484588, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) -6375> 2014-11-08 18:14:58.484699 7fc5a56ef700 5 -- op tracker -- seq: 46990, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) -6374> 2014-11-08 18:14:58.538714 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2220 ==== MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 4196083+0+0 (2709019499 0 0) 0x2393aa00 con 0x1e50aec0 -6373> 2014-11-08 18:14:58.538768 7fc5a56ef700 5 -- op tracker -- seq: 46991, time: 2014-11-08 18:14:58.484759, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6372> 2014-11-08 18:14:58.538788 7fc5a56ef700 5 -- op tracker -- seq: 46991, time: 2014-11-08 18:14:58.484760, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6371> 2014-11-08 18:14:58.538804 7fc5a56ef700 5 -- op tracker -- seq: 46991, time: 2014-11-08 18:14:58.538637, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6370> 2014-11-08 18:14:58.538819 7fc5a56ef700 5 -- op tracker -- seq: 46991, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6369> 2014-11-08 18:14:58.538870 7fc5af314700 5 -- op tracker -- seq: 46991, time: 2014-11-08 18:14:58.538870, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6368> 2014-11-08 18:14:58.577901 7fc5a56ef700 5 -- op tracker -- seq: 46991, time: 2014-11-08 18:14:58.577901, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(553e90b7/1000001a672.000000c8/head//4, version: 397'2109, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(553e90b7/1000001a672.000000c8/head//4@397'2109, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8c4e90b7/1000002b5bc.00000056/head//4, version: 3473'6548, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c4e90b7/1000002b5bc.00000056/head//4@3473'6548, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6367> 2014-11-08 18:14:58.579712 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2221 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x22456540 con 0x1e50aec0 -6366> 2014-11-08 18:14:58.579740 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.577992, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -6365> 2014-11-08 18:14:58.579745 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.577993, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -6364> 2014-11-08 18:14:58.579748 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.578061, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -6363> 2014-11-08 18:14:58.579750 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.579735, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -6362> 2014-11-08 18:14:58.579753 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.579753, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -6361> 2014-11-08 18:14:58.579761 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.579761, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -6360> 2014-11-08 18:14:58.579766 7fc5b9b29700 5 -- op tracker -- seq: 46992, time: 2014-11-08 18:14:58.579766, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -6359> 2014-11-08 18:14:58.580083 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2222 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) v2 ==== 110+0+0 (480878623 0 0) 0x2393aa00 con 0x1e50aec0 -6358> 2014-11-08 18:14:58.580097 7fc5a56ef700 5 -- op tracker -- seq: 46993, time: 2014-11-08 18:14:58.578088, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) -6357> 2014-11-08 18:14:58.580102 7fc5a56ef700 5 -- op tracker -- seq: 46993, time: 2014-11-08 18:14:58.578089, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) -6356> 2014-11-08 18:14:58.580107 7fc5a56ef700 5 -- op tracker -- seq: 46993, time: 2014-11-08 18:14:58.578123, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) -6355> 2014-11-08 18:14:58.580111 7fc5a56ef700 5 -- op tracker -- seq: 46993, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) -6354> 2014-11-08 18:14:58.580146 7fc5b0316700 5 -- op tracker -- seq: 46993, time: 2014-11-08 18:14:58.580146, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) -6353> 2014-11-08 18:14:58.580245 7fc5a56ef700 5 -- op tracker -- seq: 46993, time: 2014-11-08 18:14:58.580245, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(cd9e1d83/1000001a9f6.00000159/head//4)]) -6352> 2014-11-08 18:14:58.602431 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=75be1d83/100000265df.00000001/head//4)) v1 -- ?+0 0x1ea79680 con 0x1c5c0580 -6351> 2014-11-08 18:14:58.602470 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7fbc0 con 0x42c86e0 -6350> 2014-11-08 18:14:58.602484 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 126f1d83/1000002b7d9.00000099/head//4 -6349> 2014-11-08 18:14:58.602511 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb cd9e1d83/1000001a9f6.00000159/head//4) v3 -- ?+0 0x1ce98d80 con 0x1e50aec0 -6348> 2014-11-08 18:14:58.602541 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7fbc0 con 0x42c86e0 -6347> 2014-11-08 18:14:58.602553 7fc5b9328700 5 -- op tracker -- seq: 46994, time: 2014-11-08 18:14:58.602478, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) -6346> 2014-11-08 18:14:58.602563 7fc5b9328700 5 -- op tracker -- seq: 46994, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) -6345> 2014-11-08 18:14:58.602566 7fc5b9328700 5 -- op tracker -- seq: 46994, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) -6344> 2014-11-08 18:14:58.602568 7fc5b9328700 5 -- op tracker -- seq: 46994, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) -6343> 2014-11-08 18:14:58.602602 7fc5adb11700 5 -- op tracker -- seq: 46994, time: 2014-11-08 18:14:58.602602, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) -6342> 2014-11-08 18:14:58.658672 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) v1 -- ?+0 0x21a4c100 con 0x42c86e0 -6341> 2014-11-08 18:14:58.658724 7fc5adb11700 5 -- op tracker -- seq: 46994, time: 2014-11-08 18:14:58.658723, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4413, to_read={75be1d83/100000265df.00000001/head//4=0,4194304}, attrs_to_read=)) -6340> 2014-11-08 18:14:58.658731 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4c100 con 0x42c86e0 -6339> 2014-11-08 18:14:58.658744 7fc5b9328700 5 -- op tracker -- seq: 46995, time: 2014-11-08 18:14:58.658713, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) -6338> 2014-11-08 18:14:58.658748 7fc5b9328700 5 -- op tracker -- seq: 46995, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) -6337> 2014-11-08 18:14:58.658750 7fc5b9328700 5 -- op tracker -- seq: 46995, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) -6336> 2014-11-08 18:14:58.658752 7fc5b9328700 5 -- op tracker -- seq: 46995, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) -6335> 2014-11-08 18:14:58.658783 7fc5b0316700 5 -- op tracker -- seq: 46995, time: 2014-11-08 18:14:58.658783, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) -6334> 2014-11-08 18:14:58.658823 7fc5b0316700 5 -- op tracker -- seq: 46995, time: 2014-11-08 18:14:58.658823, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=0)) -6333> 2014-11-08 18:14:58.678485 7fc5c362f700 5 osd.9 5773 tick -6332> 2014-11-08 18:14:58.783315 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8536 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) v1 ==== 416288+0+0 (1413772914 0 0) 0x21a4b480 con 0x1c5c0580 -6331> 2014-11-08 18:14:58.783349 7fc5a152e700 5 -- op tracker -- seq: 46996, time: 2014-11-08 18:14:58.732572, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) -6330> 2014-11-08 18:14:58.783359 7fc5a152e700 5 -- op tracker -- seq: 46996, time: 2014-11-08 18:14:58.732575, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) -6329> 2014-11-08 18:14:58.783362 7fc5a152e700 5 -- op tracker -- seq: 46996, time: 2014-11-08 18:14:58.783307, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) -6328> 2014-11-08 18:14:58.783366 7fc5a152e700 5 -- op tracker -- seq: 46996, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) -6327> 2014-11-08 18:14:58.783401 7fc5adb11700 5 -- op tracker -- seq: 46996, time: 2014-11-08 18:14:58.783401, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) -6326> 2014-11-08 18:14:58.784186 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(75be1d83/100000265df.00000001/head//4, version: 3033'3548, data_included: [0~415744], data_size: 415744, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75be1d83/100000265df.00000001/head//4@3033'3548, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:831488, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393fe00 con 0x1e50aec0 -6325> 2014-11-08 18:14:58.784230 7fc5adb11700 5 -- op tracker -- seq: 46996, time: 2014-11-08 18:14:58.784230, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4413, attrs_read=1)) -6324> 2014-11-08 18:14:58.797918 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 318 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:58.793651) v2 ==== 47+0+0 (71664236 0 0) 0x239e8780 con 0x1eb9c780 -6323> 2014-11-08 18:14:58.797941 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:58.793651) v2 -- ?+0 0x2485e180 con 0x1eb9c780 -6322> 2014-11-08 18:14:58.798285 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 318 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:58.793651) v2 ==== 47+0+0 (71664236 0 0) 0x23369860 con 0x1eb9d6a0 -6321> 2014-11-08 18:14:58.798298 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:58.793651) v2 -- ?+0 0x23c31c20 con 0x1eb9d6a0 -6320> 2014-11-08 18:14:58.928219 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25190, attrs_read=0)) v1 -- ?+0 0x44a6e00 con 0x1c5c2680 -6319> 2014-11-08 18:14:58.928263 7fc5aeb13700 5 -- op tracker -- seq: 46989, time: 2014-11-08 18:14:58.928263, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25190, to_read={9a6c3227/1000002b697.0000019f/head//4=0,4194304,d8fd3227/1000001953a.0000002c/head//4=0,4194304,35ae3227/1000001a734.00000010/head//4=0,4194304,3fbe3227/1000000a440.00000000/head//4=0,4194304}, attrs_to_read=)) -6318> 2014-11-08 18:14:58.928303 7fc5aeb13700 5 -- op tracker -- seq: 46990, time: 2014-11-08 18:14:58.928303, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) -6317> 2014-11-08 18:14:58.957238 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(e71bdd13/1000001a918.00000034/head//4, version: 2829'2206, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e71bdd13/1000001a918.00000034/head//4@2829'2206, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df3cdd13/10000029ac9.0000000c/head//4, version: 3033'4387, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df3cdd13/10000029ac9.0000000c/head//4@3033'4387, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fa7ddd13/1000002a611.00000042/head//4, version: 3159'4936, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fa7ddd13/1000002a611.00000042/head//4@3159'4936, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(65addd13/10000019aef.00000433/head//4, version: 397'1020, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(65addd13/10000019aef.00000433/ head//4@397'1020, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d85a000 con 0x1c5c0580 -6316> 2014-11-08 18:14:58.964880 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(e71bdd13/1000001a918.00000034/head//4, version: 2829'2206, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e71bdd13/1000001a918.00000034/head//4@2829'2206, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(df3cdd13/10000029ac9.0000000c/head//4, version: 3033'4387, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(df3cdd13/10000029ac9.0000000c/head//4@3033'4387, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fa7ddd13/1000002a611.00000042/head//4, version: 3159'4936, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fa7ddd13/1000002a611.00000042/head//4@3159'4936, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(65addd13/10000019aef.00000433/head//4, version: 397'1020, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(65addd13/ 10000019aef.00000433/head//4@397'1020, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d858a00 con 0x1e50aec0 -6315> 2014-11-08 18:14:58.965769 7fc5aeb13700 5 -- op tracker -- seq: 46990, time: 2014-11-08 18:14:58.965769, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4410, attrs_read=4)) -6314> 2014-11-08 18:14:59.050755 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8537 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) v1 ==== 6292974+0+0 (2984264981 0 0) 0x21a4be80 con 0x1c5c0580 -6313> 2014-11-08 18:14:59.050803 7fc5a152e700 5 -- op tracker -- seq: 46997, time: 2014-11-08 18:14:58.828206, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) -6312> 2014-11-08 18:14:59.050817 7fc5a152e700 5 -- op tracker -- seq: 46997, time: 2014-11-08 18:14:58.828210, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) -6311> 2014-11-08 18:14:59.050827 7fc5a152e700 5 -- op tracker -- seq: 46997, time: 2014-11-08 18:14:59.050749, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) -6310> 2014-11-08 18:14:59.050836 7fc5a152e700 5 -- op tracker -- seq: 46997, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) -6309> 2014-11-08 18:14:59.050879 7fc5afb15700 5 -- op tracker -- seq: 46997, time: 2014-11-08 18:14:59.050879, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) -6308> 2014-11-08 18:14:59.070558 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(6dfd484/1000001abf1.00000009/head//4, version: 2829'2454, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6dfd484/1000001abf1.00000009/head//4@2829'2454, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cefd484/1000002bd64.00000049/head//4, version: 4864'8108, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cefd484/1000002bd64.00000049/head//4@4864'8108, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b2a0e484/1000002bd57.00000060/head//4, version: 4864'8090, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b2a0e484/1000002bd57.00000060/head//4@4864'8090, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3a8000 con 0x1e50aec0 -6307> 2014-11-08 18:14:59.070743 7fc5afb15700 5 -- op tracker -- seq: 46997, time: 2014-11-08 18:14:59.070743, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4412, attrs_read=3)) -6306> 2014-11-08 18:14:59.089834 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2223 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x22454560 con 0x1e50aec0 -6305> 2014-11-08 18:14:59.089862 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2224 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) v2 ==== 110+0+0 (659026261 0 0) 0x2393fe00 con 0x1e50aec0 -6304> 2014-11-08 18:14:59.089875 7fc5a56ef700 5 -- op tracker -- seq: 46999, time: 2014-11-08 18:14:59.089828, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) -6303> 2014-11-08 18:14:59.089875 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089731, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -6302> 2014-11-08 18:14:59.089890 7fc5a56ef700 5 -- op tracker -- seq: 46999, time: 2014-11-08 18:14:59.089828, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) -6301> 2014-11-08 18:14:59.089893 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089736, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -6300> 2014-11-08 18:14:59.089894 7fc5a56ef700 5 -- op tracker -- seq: 46999, time: 2014-11-08 18:14:59.089861, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) -6299> 2014-11-08 18:14:59.089897 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089812, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -6298> 2014-11-08 18:14:59.089898 7fc5a56ef700 5 -- op tracker -- seq: 46999, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) -6297> 2014-11-08 18:14:59.089899 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089869, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -6296> 2014-11-08 18:14:59.089903 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089903, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -6295> 2014-11-08 18:14:59.089909 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089909, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -6294> 2014-11-08 18:14:59.089915 7fc5b9b29700 5 -- op tracker -- seq: 46998, time: 2014-11-08 18:14:59.089915, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -6293> 2014-11-08 18:14:59.089939 7fc5b0316700 5 -- op tracker -- seq: 46999, time: 2014-11-08 18:14:59.089939, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) -6292> 2014-11-08 18:14:59.090065 7fc5b0316700 5 -- op tracker -- seq: 46999, time: 2014-11-08 18:14:59.090065, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(75be1d83/100000265df.00000001/head//4)]) -6291> 2014-11-08 18:14:59.114076 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=126f1d83/1000002b7d9.00000099/head//4)) v1 -- ?+0 0x1ea7ec00 con 0x1c5c0580 -6290> 2014-11-08 18:14:59.114113 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7de80 con 0x42c86e0 -6289> 2014-11-08 18:14:59.114126 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 5b9f1d83/1000002b56e.0000010b/head//4 -6288> 2014-11-08 18:14:59.114153 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb 75be1d83/100000265df.00000001/head//4) v3 -- ?+0 0x2482f500 con 0x1e50aec0 -6287> 2014-11-08 18:14:59.114181 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7de80 con 0x42c86e0 -6286> 2014-11-08 18:14:59.114193 7fc5b9328700 5 -- op tracker -- seq: 47000, time: 2014-11-08 18:14:59.114119, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) -6285> 2014-11-08 18:14:59.114201 7fc5b9328700 5 -- op tracker -- seq: 47000, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) -6284> 2014-11-08 18:14:59.114204 7fc5b9328700 5 -- op tracker -- seq: 47000, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) -6283> 2014-11-08 18:14:59.114207 7fc5b9328700 5 -- op tracker -- seq: 47000, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) -6282> 2014-11-08 18:14:59.114241 7fc5adb11700 5 -- op tracker -- seq: 47000, time: 2014-11-08 18:14:59.114241, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) -6281> 2014-11-08 18:14:59.173017 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) v1 -- ?+0 0x21a4b480 con 0x42c86e0 -6280> 2014-11-08 18:14:59.173066 7fc5adb11700 5 -- op tracker -- seq: 47000, time: 2014-11-08 18:14:59.173066, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4414, to_read={126f1d83/1000002b7d9.00000099/head//4=0,4194304}, attrs_to_read=)) -6279> 2014-11-08 18:14:59.173067 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a4b480 con 0x42c86e0 -6278> 2014-11-08 18:14:59.173091 7fc5b9328700 5 -- op tracker -- seq: 47001, time: 2014-11-08 18:14:59.173054, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) -6277> 2014-11-08 18:14:59.173095 7fc5b9328700 5 -- op tracker -- seq: 47001, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) -6276> 2014-11-08 18:14:59.173098 7fc5b9328700 5 -- op tracker -- seq: 47001, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) -6275> 2014-11-08 18:14:59.173099 7fc5b9328700 5 -- op tracker -- seq: 47001, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) -6274> 2014-11-08 18:14:59.173130 7fc5b0316700 5 -- op tracker -- seq: 47001, time: 2014-11-08 18:14:59.173130, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) -6273> 2014-11-08 18:14:59.173223 7fc5b0316700 5 -- op tracker -- seq: 47001, time: 2014-11-08 18:14:59.173223, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=0)) -6272> 2014-11-08 18:14:59.314274 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) v1 -- ?+0 0x21838a00 con 0x42c86e0 -6271> 2014-11-08 18:14:59.314381 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21838a00 con 0x42c86e0 -6270> 2014-11-08 18:14:59.314395 7fc5b9328700 5 -- op tracker -- seq: 47002, time: 2014-11-08 18:14:59.314372, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) -6269> 2014-11-08 18:14:59.314403 7fc5b9328700 5 -- op tracker -- seq: 47002, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) -6268> 2014-11-08 18:14:59.314405 7fc5b9328700 5 -- op tracker -- seq: 47002, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) -6267> 2014-11-08 18:14:59.314407 7fc5b9328700 5 -- op tracker -- seq: 47002, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) -6266> 2014-11-08 18:14:59.314476 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ee720 con 0x1c5c0580 -6265> 2014-11-08 18:14:59.314493 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(32ae6b16/10000029861.0000002d/head//4)]) v2 -- ?+0 0x229b9200 con 0x1c5c0580 -6264> 2014-11-08 18:14:59.314511 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x239eaee0 con 0x1e50aec0 -6263> 2014-11-08 18:14:59.314518 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1db1d640 con 0x1e50aec0 -6262> 2014-11-08 18:14:59.314525 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(553e90b7/1000001a672.000000c8/head//4),PushReplyOp(8c4e90b7/1000002b5bc.00000056/head//4)]) v2 -- ?+0 0x1d85b800 con 0x1e50aec0 -6261> 2014-11-08 18:14:59.314666 7fc5aeb13700 5 -- op tracker -- seq: 47002, time: 2014-11-08 18:14:59.314666, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) -6260> 2014-11-08 18:14:59.314738 7fc5aeb13700 5 -- op tracker -- seq: 47002, time: 2014-11-08 18:14:59.314738, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=0)) -6259> 2014-11-08 18:14:59.314811 7fc5ac30e700 5 -- op tracker -- seq: 46983, time: 2014-11-08 18:14:59.314811, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4411, to_read={8c820518/10000019a25.000003f3/head//4=0,4194304,29040518/1000002a743.00000084/head//4=0,4194304,f6240518/1000002b12f.00000140/head//4=0,4194304,37440518/1000002a55c.00000089/head//4=0,4194304,da050518/1000002a9cd.00000050/head//4=0,4194304}, attrs_to_read=)) -6258> 2014-11-08 18:14:59.341637 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8538 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) v1 ==== 10488220+0+0 (3902303035 0 0) 0x21a48500 con 0x1c5c0580 -6257> 2014-11-08 18:14:59.341681 7fc5a152e700 5 -- op tracker -- seq: 47003, time: 2014-11-08 18:14:59.060489, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) -6256> 2014-11-08 18:14:59.341708 7fc5a152e700 5 -- op tracker -- seq: 47003, time: 2014-11-08 18:14:59.060490, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) -6255> 2014-11-08 18:14:59.341717 7fc5a152e700 5 -- op tracker -- seq: 47003, time: 2014-11-08 18:14:59.341588, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) -6254> 2014-11-08 18:14:59.341727 7fc5a152e700 5 -- op tracker -- seq: 47003, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) -6253> 2014-11-08 18:14:59.341828 7fc5aeb13700 5 -- op tracker -- seq: 47003, time: 2014-11-08 18:14:59.341828, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) -6252> 2014-11-08 18:14:59.356350 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8539 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x2026fbc0 con 0x1c5c0580 -6251> 2014-11-08 18:14:59.356390 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356203, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6250> 2014-11-08 18:14:59.356398 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356209, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6249> 2014-11-08 18:14:59.356401 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356325, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6248> 2014-11-08 18:14:59.356404 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356384, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6247> 2014-11-08 18:14:59.356407 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356407, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6246> 2014-11-08 18:14:59.356415 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356415, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6245> 2014-11-08 18:14:59.356420 7fc5b9b29700 5 -- op tracker -- seq: 47004, time: 2014-11-08 18:14:59.356420, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6244> 2014-11-08 18:14:59.356432 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8540 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x2026c560 con 0x1c5c0580 -6243> 2014-11-08 18:14:59.356438 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356349, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6242> 2014-11-08 18:14:59.356441 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356350, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6241> 2014-11-08 18:14:59.356443 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356369, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6240> 2014-11-08 18:14:59.356446 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356437, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6239> 2014-11-08 18:14:59.356449 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356449, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6238> 2014-11-08 18:14:59.356452 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356452, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6237> 2014-11-08 18:14:59.356455 7fc5b9b29700 5 -- op tracker -- seq: 47005, time: 2014-11-08 18:14:59.356455, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6236> 2014-11-08 18:14:59.356461 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8541 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x20268780 con 0x1c5c0580 -6235> 2014-11-08 18:14:59.356466 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356374, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6234> 2014-11-08 18:14:59.356469 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356374, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6233> 2014-11-08 18:14:59.356501 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356392, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6232> 2014-11-08 18:14:59.356503 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356466, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6231> 2014-11-08 18:14:59.356506 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356506, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6230> 2014-11-08 18:14:59.356510 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356510, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6229> 2014-11-08 18:14:59.356513 7fc5b9b29700 5 -- op tracker -- seq: 47006, time: 2014-11-08 18:14:59.356513, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6228> 2014-11-08 18:14:59.356519 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8542 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x202690e0 con 0x1c5c0580 -6227> 2014-11-08 18:14:59.356525 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356397, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6226> 2014-11-08 18:14:59.356528 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356397, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6225> 2014-11-08 18:14:59.356530 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356414, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6224> 2014-11-08 18:14:59.356532 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356524, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6223> 2014-11-08 18:14:59.356535 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356535, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6222> 2014-11-08 18:14:59.356539 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356539, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6221> 2014-11-08 18:14:59.356542 7fc5b9b29700 5 -- op tracker -- seq: 47007, time: 2014-11-08 18:14:59.356542, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6220> 2014-11-08 18:14:59.378270 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(8c820518/10000019a25.000003f3/head//4, version: 397'1067, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8c820518/10000019a25.000003f3/head//4@397'1067, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(29040518/1000002a743.00000084/head//4, version: 3458'5434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(29040518/1000002a743.00000084/head//4@3458'5434, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6240518/1000002b12f.00000140/head//4, version: 4157'8015, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6240518/1000002b12f.00000140/head//4@4157'8015, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37440518/1000002a55c.00000089/head//4, version: 3033'5127, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37440518/1000002a55c.00000089/ head//4@3033'5127, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da050518/1000002a9cd.00000050/head//4, version: 3458'5736, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da050518/1000002a9cd.00000050/head//4@3458'5736, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d859400 con 0x1e50aec0 -6219> 2014-11-08 18:14:59.378425 7fc5aeb13700 5 -- op tracker -- seq: 47003, time: 2014-11-08 18:14:59.378424, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4411, attrs_read=5)) -6218> 2014-11-08 18:14:59.419669 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8543 ==== MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) v1 ==== 2097696+0+0 (156948602 0 0) 0x21a4f580 con 0x1c5c0580 -6217> 2014-11-08 18:14:59.419714 7fc5a152e700 5 -- op tracker -- seq: 47008, time: 2014-11-08 18:14:59.358130, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) -6216> 2014-11-08 18:14:59.419723 7fc5a152e700 5 -- op tracker -- seq: 47008, time: 2014-11-08 18:14:59.358132, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) -6215> 2014-11-08 18:14:59.419726 7fc5a152e700 5 -- op tracker -- seq: 47008, time: 2014-11-08 18:14:59.419660, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) -6214> 2014-11-08 18:14:59.419730 7fc5a152e700 5 -- op tracker -- seq: 47008, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) -6213> 2014-11-08 18:14:59.419769 7fc5adb11700 5 -- op tracker -- seq: 47008, time: 2014-11-08 18:14:59.419769, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) -6212> 2014-11-08 18:14:59.423955 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5773 [PushOp(126f1d83/1000002b7d9.00000099/head//4, version: 3473'6747, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(126f1d83/1000002b7d9.00000099/head//4@3473'6747, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x23938400 con 0x1e50aec0 -6211> 2014-11-08 18:14:59.424043 7fc5adb11700 5 -- op tracker -- seq: 47008, time: 2014-11-08 18:14:59.424042, event: done, op: MOSDECSubOpReadReply(4.83s0 5773 ECSubReadReply(tid=4414, attrs_read=1)) -6210> 2014-11-08 18:14:59.484037 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2225 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1d4f10e0 con 0x1e50aec0 -6209> 2014-11-08 18:14:59.484076 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.483584, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6208> 2014-11-08 18:14:59.484085 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.483589, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6207> 2014-11-08 18:14:59.484089 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.484012, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6206> 2014-11-08 18:14:59.484092 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.484070, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6205> 2014-11-08 18:14:59.484096 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.484096, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6204> 2014-11-08 18:14:59.484105 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.484105, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6203> 2014-11-08 18:14:59.484112 7fc5b9b29700 5 -- op tracker -- seq: 47009, time: 2014-11-08 18:14:59.484112, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6202> 2014-11-08 18:14:59.484370 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2226 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1d4f2b20 con 0x1e50aec0 -6201> 2014-11-08 18:14:59.484380 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484337, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6200> 2014-11-08 18:14:59.484383 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484338, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6199> 2014-11-08 18:14:59.484385 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484361, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6198> 2014-11-08 18:14:59.484444 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2229 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) v2 ==== 305+0+0 (1361252248 0 0) 0x1d858a00 con 0x1e50aec0 -6197> 2014-11-08 18:14:59.484457 7fc5a56ef700 5 -- op tracker -- seq: 47011, time: 2014-11-08 18:14:59.484414, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6196> 2014-11-08 18:14:59.484462 7fc5a56ef700 5 -- op tracker -- seq: 47011, time: 2014-11-08 18:14:59.484415, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6195> 2014-11-08 18:14:59.484467 7fc5a56ef700 5 -- op tracker -- seq: 47011, time: 2014-11-08 18:14:59.484443, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6194> 2014-11-08 18:14:59.484471 7fc5a56ef700 5 -- op tracker -- seq: 47011, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6193> 2014-11-08 18:14:59.484549 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484378, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6192> 2014-11-08 18:14:59.484554 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484554, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6191> 2014-11-08 18:14:59.484558 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484558, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6190> 2014-11-08 18:14:59.484562 7fc5b9b29700 5 -- op tracker -- seq: 47010, time: 2014-11-08 18:14:59.484562, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6189> 2014-11-08 18:14:59.484567 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2227 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1d4f0f00 con 0x1e50aec0 -6188> 2014-11-08 18:14:59.484574 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484368, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6187> 2014-11-08 18:14:59.484577 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484369, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6186> 2014-11-08 18:14:59.484579 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484387, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6185> 2014-11-08 18:14:59.484582 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484573, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6184> 2014-11-08 18:14:59.484585 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484585, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6183> 2014-11-08 18:14:59.484588 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484588, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6182> 2014-11-08 18:14:59.484591 7fc5b9b29700 5 -- op tracker -- seq: 47012, time: 2014-11-08 18:14:59.484591, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6181> 2014-11-08 18:14:59.484596 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2228 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1d4f21c0 con 0x1e50aec0 -6180> 2014-11-08 18:14:59.484603 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484392, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6179> 2014-11-08 18:14:59.484606 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484392, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6178> 2014-11-08 18:14:59.484609 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484410, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6177> 2014-11-08 18:14:59.484611 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484602, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6176> 2014-11-08 18:14:59.484614 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484614, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6175> 2014-11-08 18:14:59.484617 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484617, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6174> 2014-11-08 18:14:59.484620 7fc5b9b29700 5 -- op tracker -- seq: 47013, time: 2014-11-08 18:14:59.484620, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -6173> 2014-11-08 18:14:59.484638 7fc5ac30e700 5 -- op tracker -- seq: 47011, time: 2014-11-08 18:14:59.484638, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6172> 2014-11-08 18:14:59.484678 7fc5ac30e700 5 -- op tracker -- seq: 47011, time: 2014-11-08 18:14:59.484678, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6171> 2014-11-08 18:14:59.591389 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8544 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) v2 ==== 305+0+0 (3926258216 0 0) 0x229b9200 con 0x1c5c0580 -6170> 2014-11-08 18:14:59.591412 7fc5a152e700 5 -- op tracker -- seq: 47014, time: 2014-11-08 18:14:59.591318, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6169> 2014-11-08 18:14:59.591421 7fc5a152e700 5 -- op tracker -- seq: 47014, time: 2014-11-08 18:14:59.591322, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6168> 2014-11-08 18:14:59.591426 7fc5a152e700 5 -- op tracker -- seq: 47014, time: 2014-11-08 18:14:59.591383, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6167> 2014-11-08 18:14:59.591432 7fc5a152e700 5 -- op tracker -- seq: 47014, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6166> 2014-11-08 18:14:59.591461 7fc5aeb13700 5 -- op tracker -- seq: 47014, time: 2014-11-08 18:14:59.591461, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6165> 2014-11-08 18:14:59.591569 7fc5aeb13700 5 -- op tracker -- seq: 47014, time: 2014-11-08 18:14:59.591568, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(e71bdd13/1000001a918.00000034/head//4),PushReplyOp(df3cdd13/10000029ac9.0000000c/head//4),PushReplyOp(fa7ddd13/1000002a611.00000042/head//4),PushReplyOp(65addd13/10000019aef.00000433/head//4)]) -6164> 2014-11-08 18:14:59.591939 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=730edd13/1000002ab43.00000046/head//4)) v1 -- ?+0 0x1ea7a1c0 con 0x1e50aec0 -6163> 2014-11-08 18:14:59.591953 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7c800 con 0x42c86e0 -6162> 2014-11-08 18:14:59.591965 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7c800 con 0x42c86e0 -6161> 2014-11-08 18:14:59.591963 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=1 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is bd4edd13/1000002bc0a.0000001c/head//4 -6160> 2014-11-08 18:14:59.591976 7fc5b9328700 5 -- op tracker -- seq: 47015, time: 2014-11-08 18:14:59.591958, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) -6159> 2014-11-08 18:14:59.591981 7fc5b9328700 5 -- op tracker -- seq: 47015, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) -6158> 2014-11-08 18:14:59.591984 7fc5b9328700 5 -- op tracker -- seq: 47015, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) -6157> 2014-11-08 18:14:59.591986 7fc5b9328700 5 -- op tracker -- seq: 47015, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) -6156> 2014-11-08 18:14:59.591985 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb 65addd13/10000019aef.00000433/head//4) v3 -- ?+0 0x2482da00 con 0x1c5c0580 -6155> 2014-11-08 18:14:59.591996 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb 65addd13/10000019aef.00000433/head//4) v3 -- ?+0 0x24829200 con 0x1e50aec0 -6154> 2014-11-08 18:14:59.592056 7fc5ac30e700 5 -- op tracker -- seq: 47015, time: 2014-11-08 18:14:59.592056, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) -6153> 2014-11-08 18:14:59.598622 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2230 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) v3 ==== 757+0+0 (3016416033 0 0) 0x2482f500 con 0x1e50aec0 -6152> 2014-11-08 18:14:59.598652 7fc5a56ef700 5 -- op tracker -- seq: 47016, time: 2014-11-08 18:14:59.598564, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6151> 2014-11-08 18:14:59.598658 7fc5a56ef700 5 -- op tracker -- seq: 47016, time: 2014-11-08 18:14:59.598566, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6150> 2014-11-08 18:14:59.598661 7fc5a56ef700 5 -- op tracker -- seq: 47016, time: 2014-11-08 18:14:59.598620, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6149> 2014-11-08 18:14:59.598664 7fc5a56ef700 5 -- op tracker -- seq: 47016, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6148> 2014-11-08 18:14:59.598700 7fc5acb0f700 5 -- op tracker -- seq: 47016, time: 2014-11-08 18:14:59.598700, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6147> 2014-11-08 18:14:59.598711 7fc5acb0f700 5 -- op tracker -- seq: 47016, time: 2014-11-08 18:14:59.598711, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6146> 2014-11-08 18:14:59.598754 7fc5acb0f700 5 -- op tracker -- seq: 47016, time: 2014-11-08 18:14:59.598754, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb 8c4e90b7/1000002b5bc.00000056/head//4) -6145> 2014-11-08 18:14:59.617345 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpRead(4.cs1 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=71c93e0c/1000001a6bb.000000cd/head//4,8b6b3e0c/1000002a3ac.00000059/head//4,579b3e0c/10000024926.00000000/head//4)) v1 -- ?+0 0x1ea7b840 con 0x1c5c2680 -6144> 2014-11-08 18:14:59.617368 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea78900 con 0x42c86e0 -6143> 2014-11-08 18:14:59.617381 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=3 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is f9bb3e0c/1000002ad7b.00000081/head//4 -6142> 2014-11-08 18:14:59.617402 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- pg_backfill(progress 4.cs2 e 5773/5773 lb fe393e0c/1000001b50a.000000f6/head//4) v3 -- ?+0 0x2482de80 con 0x21468c60 -6141> 2014-11-08 18:14:59.617485 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea78900 con 0x42c86e0 -6140> 2014-11-08 18:14:59.617498 7fc5b9328700 5 -- op tracker -- seq: 47017, time: 2014-11-08 18:14:59.617376, event: header_read, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) -6139> 2014-11-08 18:14:59.617506 7fc5b9328700 5 -- op tracker -- seq: 47017, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) -6138> 2014-11-08 18:14:59.617510 7fc5b9328700 5 -- op tracker -- seq: 47017, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) -6137> 2014-11-08 18:14:59.617514 7fc5b9328700 5 -- op tracker -- seq: 47017, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) -6136> 2014-11-08 18:14:59.617550 7fc5ad310700 5 -- op tracker -- seq: 47017, time: 2014-11-08 18:14:59.617550, event: reached_pg, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) -6135> 2014-11-08 18:14:59.639811 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) v1 -- ?+0 0x44a1b80 con 0x42c86e0 -6134> 2014-11-08 18:14:59.639882 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x44a1b80 con 0x42c86e0 -6133> 2014-11-08 18:14:59.639897 7fc5b9328700 5 -- op tracker -- seq: 47018, time: 2014-11-08 18:14:59.639842, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) -6132> 2014-11-08 18:14:59.639909 7fc5b9328700 5 -- op tracker -- seq: 47018, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) -6131> 2014-11-08 18:14:59.639912 7fc5b9328700 5 -- op tracker -- seq: 47018, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) -6130> 2014-11-08 18:14:59.639894 7fc5ac30e700 5 -- op tracker -- seq: 47015, time: 2014-11-08 18:14:59.639894, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4415, to_read={730edd13/1000002ab43.00000046/head//4=0,4194304}, attrs_to_read=)) -6129> 2014-11-08 18:14:59.639914 7fc5b9328700 5 -- op tracker -- seq: 47018, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) -6128> 2014-11-08 18:14:59.639944 7fc5aeb13700 5 -- op tracker -- seq: 47018, time: 2014-11-08 18:14:59.639944, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) -6127> 2014-11-08 18:14:59.639981 7fc5aeb13700 5 -- op tracker -- seq: 47018, time: 2014-11-08 18:14:59.639981, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=0)) -6126> 2014-11-08 18:14:59.653396 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2231 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x1db1d640 con 0x1e50aec0 -6125> 2014-11-08 18:14:59.653434 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653297, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -6124> 2014-11-08 18:14:59.653443 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653302, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -6123> 2014-11-08 18:14:59.653446 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653374, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -6122> 2014-11-08 18:14:59.653448 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653429, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -6121> 2014-11-08 18:14:59.653451 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653451, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -6120> 2014-11-08 18:14:59.653459 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653459, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -6119> 2014-11-08 18:14:59.653465 7fc5b9b29700 5 -- op tracker -- seq: 47019, time: 2014-11-08 18:14:59.653465, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -6118> 2014-11-08 18:14:59.653463 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2234 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) v2 ==== 240+0+0 (1681304842 0 0) 0x1d85b800 con 0x1e50aec0 -6117> 2014-11-08 18:14:59.653476 7fc5a56ef700 5 -- op tracker -- seq: 47020, time: 2014-11-08 18:14:59.653437, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) -6116> 2014-11-08 18:14:59.653482 7fc5a56ef700 5 -- op tracker -- seq: 47020, time: 2014-11-08 18:14:59.653437, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) -6115> 2014-11-08 18:14:59.653486 7fc5a56ef700 5 -- op tracker -- seq: 47020, time: 2014-11-08 18:14:59.653462, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) -6114> 2014-11-08 18:14:59.653490 7fc5a56ef700 5 -- op tracker -- seq: 47020, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) -6113> 2014-11-08 18:14:59.653492 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2232 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x239eaee0 con 0x1e50aec0 -6112> 2014-11-08 18:14:59.653500 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653390, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -6111> 2014-11-08 18:14:59.653503 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653390, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -6110> 2014-11-08 18:14:59.653506 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653409, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -6109> 2014-11-08 18:14:59.653508 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653499, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -6108> 2014-11-08 18:14:59.653511 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653511, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -6107> 2014-11-08 18:14:59.653515 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653514, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -6106> 2014-11-08 18:14:59.653518 7fc5b9b29700 5 -- op tracker -- seq: 47021, time: 2014-11-08 18:14:59.653518, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -6105> 2014-11-08 18:14:59.653523 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2233 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x227ae900 con 0x1e50aec0 -6104> 2014-11-08 18:14:59.653529 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653414, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -6103> 2014-11-08 18:14:59.653532 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653414, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -6102> 2014-11-08 18:14:59.653534 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653432, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -6101> 2014-11-08 18:14:59.653537 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653528, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -6100> 2014-11-08 18:14:59.653539 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653539, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -6099> 2014-11-08 18:14:59.653543 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653543, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -6098> 2014-11-08 18:14:59.653546 7fc5b9b29700 5 -- op tracker -- seq: 47022, time: 2014-11-08 18:14:59.653546, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -6097> 2014-11-08 18:14:59.653582 7fc5afb15700 5 -- op tracker -- seq: 47020, time: 2014-11-08 18:14:59.653582, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) -6096> 2014-11-08 18:14:59.653789 7fc5afb15700 5 -- op tracker -- seq: 47020, time: 2014-11-08 18:14:59.653789, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(6dfd484/1000001abf1.00000009/head//4),PushReplyOp(1cefd484/1000002bd64.00000049/head//4),PushReplyOp(b2a0e484/1000002bd57.00000060/head//4)]) -6095> 2014-11-08 18:14:59.678581 7fc5c362f700 5 osd.9 5773 tick -6094> 2014-11-08 18:14:59.824794 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=1bd0e484/1000002aaa9.00000053/head//4,f2f0e484/1000001b4fe.00000089/head//4,7572e484/10000019cfb.00000321/head//4)) v1 -- ?+0 0x1ea7b600 con 0x1c5c0580 -6093> 2014-11-08 18:14:59.824852 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea78480 con 0x42c86e0 -6092> 2014-11-08 18:14:59.824872 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea78480 con 0x42c86e0 -6091> 2014-11-08 18:14:59.824868 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 31d2e484/1000002abe7.0000000b/head//4 -6090> 2014-11-08 18:14:59.824892 7fc5b9328700 5 -- op tracker -- seq: 47023, time: 2014-11-08 18:14:59.824862, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) -6089> 2014-11-08 18:14:59.824898 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb b2a0e484/1000002bd57.00000060/head//4) v3 -- ?+0 0x248b6300 con 0x1e50aec0 -6088> 2014-11-08 18:14:59.824903 7fc5b9328700 5 -- op tracker -- seq: 47023, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) -6087> 2014-11-08 18:14:59.824907 7fc5b9328700 5 -- op tracker -- seq: 47023, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) -6086> 2014-11-08 18:14:59.824911 7fc5b9328700 5 -- op tracker -- seq: 47023, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) -6085> 2014-11-08 18:14:59.825047 7fc5afb15700 5 -- op tracker -- seq: 47023, time: 2014-11-08 18:14:59.825047, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) -6084> 2014-11-08 18:14:59.843810 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) v1 -- ?+0 0x1d5f9900 con 0x42c86e0 -6083> 2014-11-08 18:14:59.843841 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1d5f9900 con 0x42c86e0 -6082> 2014-11-08 18:14:59.843841 7fc5ad310700 5 -- op tracker -- seq: 47017, time: 2014-11-08 18:14:59.843841, event: done, op: MOSDECSubOpRead(4.cs0 5773 ECSubRead(tid=4416, to_read={71c93e0c/1000001a6bb.000000cd/head//4=0,4194304,8b6b3e0c/1000002a3ac.00000059/head//4=0,4194304,579b3e0c/10000024926.00000000/head//4=0,4194304}, attrs_to_read=)) -6081> 2014-11-08 18:14:59.843856 7fc5b9328700 5 -- op tracker -- seq: 47024, time: 2014-11-08 18:14:59.843832, event: header_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) -6080> 2014-11-08 18:14:59.843860 7fc5b9328700 5 -- op tracker -- seq: 47024, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) -6079> 2014-11-08 18:14:59.843862 7fc5b9328700 5 -- op tracker -- seq: 47024, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) -6078> 2014-11-08 18:14:59.843864 7fc5b9328700 5 -- op tracker -- seq: 47024, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) -6077> 2014-11-08 18:14:59.843886 7fc5ad310700 5 -- op tracker -- seq: 47024, time: 2014-11-08 18:14:59.843886, event: reached_pg, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) -6076> 2014-11-08 18:14:59.843920 7fc5ad310700 5 -- op tracker -- seq: 47024, time: 2014-11-08 18:14:59.843920, event: done, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=0)) -6075> 2014-11-08 18:14:59.905410 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3677 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:59.902998) v2 ==== 47+0+0 (2771925885 0 0) 0x224a2d00 con 0x1cce2ec0 -6074> 2014-11-08 18:14:59.905439 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:59.902998) v2 -- ?+0 0x23369860 con 0x1cce2ec0 -6073> 2014-11-08 18:14:59.905506 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3677 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:59.902998) v2 ==== 47+0+0 (2771925885 0 0) 0x23140780 con 0x1cce3440 -6072> 2014-11-08 18:14:59.905519 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:59.902998) v2 -- ?+0 0x239e8780 con 0x1cce3440 -6071> 2014-11-08 18:14:59.922162 7fc5b2b1b700 10 monclient: tick -6070> 2014-11-08 18:14:59.922190 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:14:29.922188) -6069> 2014-11-08 18:14:59.922208 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:14:59.922208; renew after: 2014-11-08 18:16:29.290049) -- no -6068> 2014-11-08 18:15:00.001484 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 498 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:59.999130) v2 ==== 47+0+0 (770878784 0 0) 0x1e6afbc0 con 0x1eb9c8e0 -6067> 2014-11-08 18:15:00.001513 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:59.999130) v2 -- ?+0 0x23140780 con 0x1eb9c8e0 -6066> 2014-11-08 18:15:00.001716 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 498 ==== osd_ping(ping e5773 stamp 2014-11-08 18:14:59.999130) v2 ==== 47+0+0 (770878784 0 0) 0x1e6adfa0 con 0x1eb9d800 -6065> 2014-11-08 18:15:00.001729 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:14:59.999130) v2 -- ?+0 0x224a2d00 con 0x1eb9d800 -6064> 2014-11-08 18:15:00.119994 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) v1 -- ?+0 0x224baf80 con 0x42c86e0 -6063> 2014-11-08 18:15:00.120049 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224baf80 con 0x42c86e0 -6062> 2014-11-08 18:15:00.120066 7fc5b9328700 5 -- op tracker -- seq: 47025, time: 2014-11-08 18:15:00.120033, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) -6061> 2014-11-08 18:15:00.120077 7fc5b9328700 5 -- op tracker -- seq: 47025, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) -6060> 2014-11-08 18:15:00.120080 7fc5b9328700 5 -- op tracker -- seq: 47025, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) -6059> 2014-11-08 18:15:00.120081 7fc5b9328700 5 -- op tracker -- seq: 47025, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) -6058> 2014-11-08 18:15:00.120066 7fc5afb15700 5 -- op tracker -- seq: 47023, time: 2014-11-08 18:15:00.120066, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4417, to_read={1bd0e484/1000002aaa9.00000053/head//4=0,4194304,f2f0e484/1000001b4fe.00000089/head//4=0,4194304,7572e484/10000019cfb.00000321/head//4=0,4194304}, attrs_to_read=)) -6057> 2014-11-08 18:15:00.120120 7fc5ad310700 5 -- op tracker -- seq: 47025, time: 2014-11-08 18:15:00.120120, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) -6056> 2014-11-08 18:15:00.120167 7fc5ad310700 5 -- op tracker -- seq: 47025, time: 2014-11-08 18:15:00.120167, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=0)) -6055> 2014-11-08 18:15:00.318792 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3741 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:00.318375) v2 ==== 47+0+0 (3120891520 0 0) 0x23c33c00 con 0x1cce3180 -6054> 2014-11-08 18:15:00.318824 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:00.318375) v2 -- ?+0 0x1e6adfa0 con 0x1cce3180 -6053> 2014-11-08 18:15:00.318853 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3741 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:00.318375) v2 ==== 47+0+0 (3120891520 0 0) 0x2485f440 con 0x1cce3700 -6052> 2014-11-08 18:15:00.318862 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:00.318375) v2 -- ?+0 0x1e6afbc0 con 0x1cce3700 -6051> 2014-11-08 18:15:00.367359 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2235 ==== MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2867317756 0 0) 0x23938400 con 0x1e50aec0 -6050> 2014-11-08 18:15:00.367400 7fc5a56ef700 5 -- op tracker -- seq: 47026, time: 2014-11-08 18:15:00.334192, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6049> 2014-11-08 18:15:00.367412 7fc5a56ef700 5 -- op tracker -- seq: 47026, time: 2014-11-08 18:15:00.334195, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6048> 2014-11-08 18:15:00.367419 7fc5a56ef700 5 -- op tracker -- seq: 47026, time: 2014-11-08 18:15:00.367351, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6047> 2014-11-08 18:15:00.367424 7fc5a56ef700 5 -- op tracker -- seq: 47026, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6046> 2014-11-08 18:15:00.367465 7fc5af314700 5 -- op tracker -- seq: 47026, time: 2014-11-08 18:15:00.367464, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6045> 2014-11-08 18:15:00.367560 7fc5af314700 5 -- op tracker -- seq: 47026, time: 2014-11-08 18:15:00.367560, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(c8ae90b7/1000001aac4.00000043/head//4, version: 2829'2489, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8ae90b7/1000001aac4.00000043/head//4@2829'2489, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -6044> 2014-11-08 18:15:00.387987 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2236 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) v1 ==== 2097696+0+0 (2297562336 0 0) 0x224bd500 con 0x1e50aec0 -6043> 2014-11-08 18:15:00.388023 7fc5a56ef700 5 -- op tracker -- seq: 47027, time: 2014-11-08 18:15:00.367458, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) -6042> 2014-11-08 18:15:00.388033 7fc5a56ef700 5 -- op tracker -- seq: 47027, time: 2014-11-08 18:15:00.367459, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) -6041> 2014-11-08 18:15:00.388036 7fc5a56ef700 5 -- op tracker -- seq: 47027, time: 2014-11-08 18:15:00.387976, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) -6040> 2014-11-08 18:15:00.388040 7fc5a56ef700 5 -- op tracker -- seq: 47027, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) -6039> 2014-11-08 18:15:00.388079 7fc5aeb13700 5 -- op tracker -- seq: 47027, time: 2014-11-08 18:15:00.388078, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) -6038> 2014-11-08 18:15:00.392890 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(730edd13/1000002ab43.00000046/head//4, version: 3467'5644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(730edd13/1000002ab43.00000046/head//4@3467'5644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9200 con 0x1c5c0580 -6037> 2014-11-08 18:15:00.393875 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(730edd13/1000002ab43.00000046/head//4, version: 3467'5644, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(730edd13/1000002ab43.00000046/head//4@3467'5644, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d85f600 con 0x1e50aec0 -6036> 2014-11-08 18:15:00.393921 7fc5aeb13700 5 -- op tracker -- seq: 47027, time: 2014-11-08 18:15:00.393920, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4415, attrs_read=1)) -6035> 2014-11-08 18:15:00.428898 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1db1f440 con 0x1e50aec0 -6034> 2014-11-08 18:15:00.437903 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(c8ae90b7/1000001aac4.00000043/head//4)]) v2 -- ?+0 0x21617200 con 0x1e50aec0 -6033> 2014-11-08 18:15:00.503337 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 499 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:00.499892) v2 ==== 47+0+0 (484910732 0 0) 0x1e6addc0 con 0x1eb9d800 -6032> 2014-11-08 18:15:00.503365 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:00.499892) v2 -- ?+0 0x23c33c00 con 0x1eb9d800 -6031> 2014-11-08 18:15:00.503391 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 499 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:00.499892) v2 ==== 47+0+0 (484910732 0 0) 0x1e6af9e0 con 0x1eb9c8e0 -6030> 2014-11-08 18:15:00.503402 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:00.499892) v2 -- ?+0 0x2485f440 con 0x1eb9c8e0 -6029> 2014-11-08 18:15:00.618031 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12061 ==== MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) v1 ==== 4202195+0+0 (3697767910 0 0) 0x44a6e00 con 0x1c5c2680 -6028> 2014-11-08 18:15:00.618067 7fc5a6cfe700 5 -- op tracker -- seq: 47028, time: 2014-11-08 18:15:00.279043, event: header_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) -6027> 2014-11-08 18:15:00.618081 7fc5a6cfe700 5 -- op tracker -- seq: 47028, time: 2014-11-08 18:15:00.279048, event: throttled, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) -6026> 2014-11-08 18:15:00.618090 7fc5a6cfe700 5 -- op tracker -- seq: 47028, time: 2014-11-08 18:15:00.618000, event: all_read, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) -6025> 2014-11-08 18:15:00.618099 7fc5a6cfe700 5 -- op tracker -- seq: 47028, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) -6024> 2014-11-08 18:15:00.618165 7fc5afb15700 5 -- op tracker -- seq: 47028, time: 2014-11-08 18:15:00.618165, event: reached_pg, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) -6023> 2014-11-08 18:15:00.627369 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDPGPush(4.cs2 5773 [PushOp(71c93e0c/1000001a6bb.000000cd/head//4, version: 397'2001, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(71c93e0c/1000001a6bb.000000cd/head//4@397'2001, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8b6b3e0c/1000002a3ac.00000059/head//4, version: 3033'4646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8b6b3e0c/1000002a3ac.00000059/head//4@3033'4646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(579b3e0c/10000024926.00000000/head//4, version: 3033'3379, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(579b3e0c/10000024926.00000000/head//4@3033'3379, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d85b800 con 0x21468c60 -6022> 2014-11-08 18:15:00.627486 7fc5afb15700 5 -- op tracker -- seq: 47028, time: 2014-11-08 18:15:00.627485, event: done, op: MOSDECSubOpReadReply(4.cs0 5773 ECSubReadReply(tid=4416, attrs_read=3)) -6021> 2014-11-08 18:15:00.674318 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8545 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) v1 ==== 6292958+0+0 (4104669388 0 0) 0x21a48a00 con 0x1c5c0580 -6020> 2014-11-08 18:15:00.674348 7fc5a152e700 5 -- op tracker -- seq: 47029, time: 2014-11-08 18:15:00.175466, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) -6019> 2014-11-08 18:15:00.674356 7fc5a152e700 5 -- op tracker -- seq: 47029, time: 2014-11-08 18:15:00.175471, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) -6018> 2014-11-08 18:15:00.674359 7fc5a152e700 5 -- op tracker -- seq: 47029, time: 2014-11-08 18:15:00.674294, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) -6017> 2014-11-08 18:15:00.674362 7fc5a152e700 5 -- op tracker -- seq: 47029, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) -6016> 2014-11-08 18:15:00.674399 7fc5ad310700 5 -- op tracker -- seq: 47029, time: 2014-11-08 18:15:00.674399, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) -6015> 2014-11-08 18:15:00.678710 7fc5c362f700 5 osd.9 5773 tick -6014> 2014-11-08 18:15:00.678756 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -6013> 2014-11-08 18:15:00.678767 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(4 pgs tid 1329 v 0) v1 -- ?+0 0x248af900 con 0x44ed6a0 -6012> 2014-11-08 18:15:00.682815 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8546 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) v3 ==== 757+0+0 (99203210 0 0) 0x1ce9b600 con 0x1c5c0580 -6011> 2014-11-08 18:15:00.682849 7fc5a152e700 5 -- op tracker -- seq: 47030, time: 2014-11-08 18:15:00.682657, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -6010> 2014-11-08 18:15:00.682860 7fc5a152e700 5 -- op tracker -- seq: 47030, time: 2014-11-08 18:15:00.682659, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -6009> 2014-11-08 18:15:00.682865 7fc5a152e700 5 -- op tracker -- seq: 47030, time: 2014-11-08 18:15:00.682793, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -6008> 2014-11-08 18:15:00.682902 7fc5a152e700 5 -- op tracker -- seq: 47030, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -6007> 2014-11-08 18:15:00.682970 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8547 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x239ee720 con 0x1c5c0580 -6006> 2014-11-08 18:15:00.683013 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.682930, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -6005> 2014-11-08 18:15:00.683018 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.682931, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -6004> 2014-11-08 18:15:00.683023 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.682952, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -6003> 2014-11-08 18:15:00.683028 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.683009, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -6002> 2014-11-08 18:15:00.683033 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.683033, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -6001> 2014-11-08 18:15:00.683040 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.683040, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -6000> 2014-11-08 18:15:00.683036 7fc5afb15700 5 -- op tracker -- seq: 47030, time: 2014-11-08 18:15:00.683036, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -5999> 2014-11-08 18:15:00.683047 7fc5b9b29700 5 -- op tracker -- seq: 47031, time: 2014-11-08 18:15:00.683047, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -5998> 2014-11-08 18:15:00.683057 7fc5afb15700 5 -- op tracker -- seq: 47030, time: 2014-11-08 18:15:00.683057, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -5997> 2014-11-08 18:15:00.683114 7fc5afb15700 5 -- op tracker -- seq: 47030, time: 2014-11-08 18:15:00.683114, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 32ae6b16/10000029861.0000002d/head//4) -5996> 2014-11-08 18:15:00.683107 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8548 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) v2 ==== 110+0+0 (3005507917 0 0) 0x229b9200 con 0x1c5c0580 -5995> 2014-11-08 18:15:00.683357 7fc5a152e700 5 -- op tracker -- seq: 47032, time: 2014-11-08 18:15:00.683070, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5994> 2014-11-08 18:15:00.683363 7fc5a152e700 5 -- op tracker -- seq: 47032, time: 2014-11-08 18:15:00.683071, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5993> 2014-11-08 18:15:00.683368 7fc5a152e700 5 -- op tracker -- seq: 47032, time: 2014-11-08 18:15:00.683098, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5992> 2014-11-08 18:15:00.683373 7fc5a152e700 5 -- op tracker -- seq: 47032, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5991> 2014-11-08 18:15:00.683396 7fc5ac30e700 5 -- op tracker -- seq: 47032, time: 2014-11-08 18:15:00.683396, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5990> 2014-11-08 18:15:00.683429 7fc5a152e700 5 -- op tracker -- seq: 47032, time: 2014-11-08 18:15:00.683429, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5989> 2014-11-08 18:15:00.690030 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(1bd0e484/1000002aaa9.00000053/head//4, version: 3458'5717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1bd0e484/1000002aaa9.00000053/head//4@3458'5717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2f0e484/1000001b4fe.00000089/head//4, version: 3033'3126, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2f0e484/1000001b4fe.00000089/head//4@3033'3126, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7572e484/10000019cfb.00000321/head//4, version: 397'1248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7572e484/10000019cfb.00000321/head//4@397'1248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x24840c00 con 0x1e50aec0 -5988> 2014-11-08 18:15:00.693987 7fc5ad310700 5 -- op tracker -- seq: 47029, time: 2014-11-08 18:15:00.693986, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4417, attrs_read=3)) -5987> 2014-11-08 18:15:00.697980 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2237 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1db1f440 con 0x1e50aec0 -5986> 2014-11-08 18:15:00.698027 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.697870, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5985> 2014-11-08 18:15:00.698037 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.697876, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5984> 2014-11-08 18:15:00.698040 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.697953, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5983> 2014-11-08 18:15:00.698042 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.698021, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5982> 2014-11-08 18:15:00.698046 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.698046, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5981> 2014-11-08 18:15:00.698057 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.698057, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5980> 2014-11-08 18:15:00.698063 7fc5b9b29700 5 -- op tracker -- seq: 47033, time: 2014-11-08 18:15:00.698063, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5979> 2014-11-08 18:15:00.699194 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2238 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1d4f0000 con 0x1e50aec0 -5978> 2014-11-08 18:15:00.699220 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699079, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5977> 2014-11-08 18:15:00.699226 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699083, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5976> 2014-11-08 18:15:00.699229 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699177, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5975> 2014-11-08 18:15:00.699231 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699216, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5974> 2014-11-08 18:15:00.699234 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699234, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5973> 2014-11-08 18:15:00.699241 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699241, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5972> 2014-11-08 18:15:00.699245 7fc5b9b29700 5 -- op tracker -- seq: 47034, time: 2014-11-08 18:15:00.699245, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5971> 2014-11-08 18:15:00.699253 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2239 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1d4f10e0 con 0x1e50aec0 -5970> 2014-11-08 18:15:00.699260 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699188, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5969> 2014-11-08 18:15:00.699263 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699189, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5968> 2014-11-08 18:15:00.699265 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699208, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5967> 2014-11-08 18:15:00.699268 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699259, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5966> 2014-11-08 18:15:00.699271 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699271, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5965> 2014-11-08 18:15:00.699274 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699274, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5964> 2014-11-08 18:15:00.699277 7fc5b9b29700 5 -- op tracker -- seq: 47035, time: 2014-11-08 18:15:00.699277, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5963> 2014-11-08 18:15:00.699282 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2240 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x234e6cc0 con 0x1e50aec0 -5962> 2014-11-08 18:15:00.699288 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699213, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5961> 2014-11-08 18:15:00.699291 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699214, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5960> 2014-11-08 18:15:00.699293 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699231, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5959> 2014-11-08 18:15:00.699295 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699287, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5958> 2014-11-08 18:15:00.699298 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699297, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5957> 2014-11-08 18:15:00.699301 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699301, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5956> 2014-11-08 18:15:00.699304 7fc5b9b29700 5 -- op tracker -- seq: 47036, time: 2014-11-08 18:15:00.699304, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5955> 2014-11-08 18:15:00.699309 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2241 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x234e6360 con 0x1e50aec0 -5954> 2014-11-08 18:15:00.699315 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699236, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5953> 2014-11-08 18:15:00.699318 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699236, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5952> 2014-11-08 18:15:00.699320 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699254, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5951> 2014-11-08 18:15:00.699323 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699314, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5950> 2014-11-08 18:15:00.699326 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699326, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5949> 2014-11-08 18:15:00.699329 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699329, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5948> 2014-11-08 18:15:00.699332 7fc5b9b29700 5 -- op tracker -- seq: 47037, time: 2014-11-08 18:15:00.699332, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5947> 2014-11-08 18:15:00.699300 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2242 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) v2 ==== 370+0+0 (1551438691 0 0) 0x21617200 con 0x1e50aec0 -5946> 2014-11-08 18:15:00.699359 7fc5a56ef700 5 -- op tracker -- seq: 47038, time: 2014-11-08 18:15:00.699259, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) -5945> 2014-11-08 18:15:00.699365 7fc5a56ef700 5 -- op tracker -- seq: 47038, time: 2014-11-08 18:15:00.699259, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) -5944> 2014-11-08 18:15:00.699370 7fc5a56ef700 5 -- op tracker -- seq: 47038, time: 2014-11-08 18:15:00.699299, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) -5943> 2014-11-08 18:15:00.699375 7fc5a56ef700 5 -- op tracker -- seq: 47038, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) -5942> 2014-11-08 18:15:00.699411 7fc5aeb13700 5 -- op tracker -- seq: 47038, time: 2014-11-08 18:15:00.699411, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) -5941> 2014-11-08 18:15:00.699477 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2244 ==== MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) v2 ==== 110+0+0 (390557927 0 0) 0x1d85f600 con 0x1e50aec0 -5940> 2014-11-08 18:15:00.699485 7fc5a56ef700 5 -- op tracker -- seq: 47039, time: 2014-11-08 18:15:00.699429, event: header_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) -5939> 2014-11-08 18:15:00.699489 7fc5a56ef700 5 -- op tracker -- seq: 47039, time: 2014-11-08 18:15:00.699429, event: throttled, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) -5938> 2014-11-08 18:15:00.699492 7fc5a56ef700 5 -- op tracker -- seq: 47039, time: 2014-11-08 18:15:00.699476, event: all_read, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) -5937> 2014-11-08 18:15:00.699495 7fc5a56ef700 5 -- op tracker -- seq: 47039, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) -5936> 2014-11-08 18:15:00.699533 7fc5aeb13700 5 -- op tracker -- seq: 47038, time: 2014-11-08 18:15:00.699533, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(8c820518/10000019a25.000003f3/head//4),PushReplyOp(29040518/1000002a743.00000084/head//4),PushReplyOp(f6240518/1000002b12f.00000140/head//4),PushReplyOp(37440518/1000002a55c.00000089/head//4),PushReplyOp(da050518/1000002a9cd.00000050/head//4)]) -5935> 2014-11-08 18:15:00.699556 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2246 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) v2 ==== 110+0+0 (134386941 0 0) 0x1d859400 con 0x1e50aec0 -5934> 2014-11-08 18:15:00.699564 7fc5a56ef700 5 -- op tracker -- seq: 47040, time: 2014-11-08 18:15:00.699535, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5933> 2014-11-08 18:15:00.699567 7fc5a56ef700 5 -- op tracker -- seq: 47040, time: 2014-11-08 18:15:00.699535, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5932> 2014-11-08 18:15:00.699570 7fc5a56ef700 5 -- op tracker -- seq: 47040, time: 2014-11-08 18:15:00.699555, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5931> 2014-11-08 18:15:00.699573 7fc5a56ef700 5 -- op tracker -- seq: 47040, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5930> 2014-11-08 18:15:00.699554 7fc5b0316700 5 -- op tracker -- seq: 47039, time: 2014-11-08 18:15:00.699554, event: reached_pg, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) -5929> 2014-11-08 18:15:00.699620 7fc5b0316700 5 -- op tracker -- seq: 47039, time: 2014-11-08 18:15:00.699620, event: done, op: MOSDPGPushReply(4.83s0 5773 [PushReplyOp(126f1d83/1000002b7d9.00000099/head//4)]) -5928> 2014-11-08 18:15:00.699632 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2243 ==== pg_trim(4.83 to 5734'8135 e5773) v1 ==== 34+0+0 (3351629388 0 0) 0x234e2d00 con 0x1e50aec0 -5927> 2014-11-08 18:15:00.699640 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699401, event: header_read, op: pg_trim(4.83 to 5734'8135 e5773) -5926> 2014-11-08 18:15:00.699643 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699401, event: throttled, op: pg_trim(4.83 to 5734'8135 e5773) -5925> 2014-11-08 18:15:00.699646 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699421, event: all_read, op: pg_trim(4.83 to 5734'8135 e5773) -5924> 2014-11-08 18:15:00.699649 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699639, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5773) -5923> 2014-11-08 18:15:00.699651 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699651, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5773) -5922> 2014-11-08 18:15:00.699655 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699655, event: started, op: pg_trim(4.83 to 5734'8135 e5773) -5921> 2014-11-08 18:15:00.699659 7fc5b9b29700 5 -- op tracker -- seq: 47041, time: 2014-11-08 18:15:00.699659, event: done, op: pg_trim(4.83 to 5734'8135 e5773) -5920> 2014-11-08 18:15:00.699708 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2245 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x234e14a0 con 0x1e50aec0 -5919> 2014-11-08 18:15:00.699715 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699511, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -5918> 2014-11-08 18:15:00.699718 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699511, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -5917> 2014-11-08 18:15:00.699717 7fc5ac30e700 5 -- op tracker -- seq: 47040, time: 2014-11-08 18:15:00.699717, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5916> 2014-11-08 18:15:00.699721 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699531, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -5915> 2014-11-08 18:15:00.699723 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699714, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -5914> 2014-11-08 18:15:00.699726 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699726, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -5913> 2014-11-08 18:15:00.699729 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699729, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -5912> 2014-11-08 18:15:00.699733 7fc5b9b29700 5 -- op tracker -- seq: 47042, time: 2014-11-08 18:15:00.699733, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -5911> 2014-11-08 18:15:00.699841 7fc5ac30e700 5 -- op tracker -- seq: 47040, time: 2014-11-08 18:15:00.699841, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(730edd13/1000002ab43.00000046/head//4)]) -5910> 2014-11-08 18:15:00.720218 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2247 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) v3 ==== 757+0+0 (4175872848 0 0) 0x1ce9cc80 con 0x1e50aec0 -5909> 2014-11-08 18:15:00.720251 7fc5a56ef700 5 -- op tracker -- seq: 47043, time: 2014-11-08 18:15:00.720139, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5908> 2014-11-08 18:15:00.720259 7fc5a56ef700 5 -- op tracker -- seq: 47043, time: 2014-11-08 18:15:00.720142, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5907> 2014-11-08 18:15:00.720262 7fc5a56ef700 5 -- op tracker -- seq: 47043, time: 2014-11-08 18:15:00.720215, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5906> 2014-11-08 18:15:00.720265 7fc5a56ef700 5 -- op tracker -- seq: 47043, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5905> 2014-11-08 18:15:00.720349 7fc5acb0f700 5 -- op tracker -- seq: 47043, time: 2014-11-08 18:15:00.720349, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5904> 2014-11-08 18:15:00.720360 7fc5acb0f700 5 -- op tracker -- seq: 47043, time: 2014-11-08 18:15:00.720360, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5903> 2014-11-08 18:15:00.720403 7fc5acb0f700 5 -- op tracker -- seq: 47043, time: 2014-11-08 18:15:00.720403, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb c8ae90b7/1000001aac4.00000043/head//4) -5902> 2014-11-08 18:15:00.823251 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=44260518/1000002ad5b.00000044/head//4,75f60518/1000001a0eb.00000616/head//4,cac70518/10000016074.00000000/head//4,9e080518/1000001a2e2.00000011/head//4,22c80518/1000002b9f1.0000000d/head//4)) v1 -- ?+0 0x1ea7e300 con 0x1c5c0580 -5901> 2014-11-08 18:15:00.823286 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7da00 con 0x42c86e0 -5900> 2014-11-08 18:15:00.823304 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is fa1c0518/1000001a653.00000126/head//4 -5899> 2014-11-08 18:15:00.823331 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb da050518/1000002a9cd.00000050/head//4) v3 -- ?+0 0x1d3c8d80 con 0x1e50aec0 -5898> 2014-11-08 18:15:00.823344 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7da00 con 0x42c86e0 -5897> 2014-11-08 18:15:00.823361 7fc5b9328700 5 -- op tracker -- seq: 47044, time: 2014-11-08 18:15:00.823297, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) -5896> 2014-11-08 18:15:00.823372 7fc5b9328700 5 -- op tracker -- seq: 47044, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) -5895> 2014-11-08 18:15:00.823377 7fc5b9328700 5 -- op tracker -- seq: 47044, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) -5894> 2014-11-08 18:15:00.823382 7fc5b9328700 5 -- op tracker -- seq: 47044, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) -5893> 2014-11-08 18:15:00.823537 7fc5aeb13700 5 -- op tracker -- seq: 47044, time: 2014-11-08 18:15:00.823537, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) -5892> 2014-11-08 18:15:00.823669 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=bd4edd13/1000002bc0a.0000001c/head//4,7e7edd13/1000002b5ea.000004c4/head//4)) v1 -- ?+0 0x1ea7b180 con 0x1e50aec0 -5891> 2014-11-08 18:15:00.823681 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7e0c0 con 0x42c86e0 -5890> 2014-11-08 18:15:00.823698 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=2 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is be8fdd13/1000002b2f6.00000008/head//4 -5889> 2014-11-08 18:15:00.823714 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb 730edd13/1000002ab43.00000046/head//4) v3 -- ?+0 0x22767080 con 0x1c5c0580 -5888> 2014-11-08 18:15:00.823724 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb 730edd13/1000002ab43.00000046/head//4) v3 -- ?+0 0x1d653600 con 0x1e50aec0 -5887> 2014-11-08 18:15:00.823841 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7e0c0 con 0x42c86e0 -5886> 2014-11-08 18:15:00.823851 7fc5b9328700 5 -- op tracker -- seq: 47045, time: 2014-11-08 18:15:00.823694, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) -5885> 2014-11-08 18:15:00.823857 7fc5b9328700 5 -- op tracker -- seq: 47045, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) -5884> 2014-11-08 18:15:00.823860 7fc5b9328700 5 -- op tracker -- seq: 47045, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) -5883> 2014-11-08 18:15:00.823863 7fc5b9328700 5 -- op tracker -- seq: 47045, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) -5882> 2014-11-08 18:15:00.823884 7fc5ac30e700 5 -- op tracker -- seq: 47045, time: 2014-11-08 18:15:00.823884, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) -5881> 2014-11-08 18:15:00.994808 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) v1 -- ?+0 0x224b8280 con 0x42c86e0 -5880> 2014-11-08 18:15:00.994842 7fc5ac30e700 5 -- op tracker -- seq: 47045, time: 2014-11-08 18:15:00.994841, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4419, to_read={bd4edd13/1000002bc0a.0000001c/head//4=0,4194304,7e7edd13/1000002b5ea.000004c4/head//4=0,4194304}, attrs_to_read=)) -5879> 2014-11-08 18:15:00.994844 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224b8280 con 0x42c86e0 -5878> 2014-11-08 18:15:00.994863 7fc5b9328700 5 -- op tracker -- seq: 47046, time: 2014-11-08 18:15:00.994831, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) -5877> 2014-11-08 18:15:00.994867 7fc5b9328700 5 -- op tracker -- seq: 47046, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) -5876> 2014-11-08 18:15:00.994869 7fc5b9328700 5 -- op tracker -- seq: 47046, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) -5875> 2014-11-08 18:15:00.994871 7fc5b9328700 5 -- op tracker -- seq: 47046, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) -5874> 2014-11-08 18:15:00.994898 7fc5ac30e700 5 -- op tracker -- seq: 47046, time: 2014-11-08 18:15:00.994898, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) -5873> 2014-11-08 18:15:00.994939 7fc5ac30e700 5 -- op tracker -- seq: 47046, time: 2014-11-08 18:15:00.994939, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=0)) -5872> 2014-11-08 18:15:01.007027 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3678 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:01.003832) v2 ==== 47+0+0 (3137632190 0 0) 0x23145dc0 con 0x1cce3440 -5871> 2014-11-08 18:15:01.007049 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:01.003832) v2 -- ?+0 0x1e6af9e0 con 0x1cce3440 -5870> 2014-11-08 18:15:01.007071 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3678 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:01.003832) v2 ==== 47+0+0 (3137632190 0 0) 0x224a5280 con 0x1cce2ec0 -5869> 2014-11-08 18:15:01.007082 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:01.003832) v2 -- ?+0 0x1e6addc0 con 0x1cce2ec0 -5868> 2014-11-08 18:15:01.096528 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 319 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:01.094320) v2 ==== 47+0+0 (949507181 0 0) 0x2336f800 con 0x1eb9d6a0 -5867> 2014-11-08 18:15:01.096601 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:01.094320) v2 -- ?+0 0x224a5280 con 0x1eb9d6a0 -5866> 2014-11-08 18:15:01.096683 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 319 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:01.094320) v2 ==== 47+0+0 (949507181 0 0) 0x239ea3a0 con 0x1eb9c780 -5865> 2014-11-08 18:15:01.096705 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:01.094320) v2 -- ?+0 0x23145dc0 con 0x1eb9c780 -5864> 2014-11-08 18:15:01.222767 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) v1 -- ?+0 0x224bd500 con 0x42c86e0 -5863> 2014-11-08 18:15:01.222820 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224bd500 con 0x42c86e0 -5862> 2014-11-08 18:15:01.222837 7fc5b9328700 5 -- op tracker -- seq: 47047, time: 2014-11-08 18:15:01.222803, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) -5861> 2014-11-08 18:15:01.222845 7fc5b9328700 5 -- op tracker -- seq: 47047, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) -5860> 2014-11-08 18:15:01.222847 7fc5b9328700 5 -- op tracker -- seq: 47047, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) -5859> 2014-11-08 18:15:01.222849 7fc5b9328700 5 -- op tracker -- seq: 47047, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) -5858> 2014-11-08 18:15:01.222875 7fc5aeb13700 5 -- op tracker -- seq: 47044, time: 2014-11-08 18:15:01.222875, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4418, to_read={44260518/1000002ad5b.00000044/head//4=0,4194304,75f60518/1000001a0eb.00000616/head//4=0,4194304,cac70518/10000016074.00000000/head//4=0,4194304,9e080518/1000001a2e2.00000011/head//4=0,4194304,22c80518/1000002b9f1.0000000d/head//4=0,4194304}, attrs_to_read=)) -5857> 2014-11-08 18:15:01.222887 7fc5ac30e700 5 -- op tracker -- seq: 47047, time: 2014-11-08 18:15:01.222887, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) -5856> 2014-11-08 18:15:01.222964 7fc5ac30e700 5 -- op tracker -- seq: 47047, time: 2014-11-08 18:15:01.222964, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=0)) -5855> 2014-11-08 18:15:01.276656 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2248 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x234e0780 con 0x1e50aec0 -5854> 2014-11-08 18:15:01.276680 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276485, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -5853> 2014-11-08 18:15:01.276692 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276487, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -5852> 2014-11-08 18:15:01.276695 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276585, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -5851> 2014-11-08 18:15:01.276698 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276676, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -5850> 2014-11-08 18:15:01.276701 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276701, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -5849> 2014-11-08 18:15:01.276708 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276708, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -5848> 2014-11-08 18:15:01.276713 7fc5b9b29700 5 -- op tracker -- seq: 47048, time: 2014-11-08 18:15:01.276713, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -5847> 2014-11-08 18:15:01.276722 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2249 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x234e4920 con 0x1e50aec0 -5846> 2014-11-08 18:15:01.276728 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276601, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -5845> 2014-11-08 18:15:01.276731 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276602, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -5844> 2014-11-08 18:15:01.276733 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276621, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -5843> 2014-11-08 18:15:01.276735 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276727, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -5842> 2014-11-08 18:15:01.276738 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276738, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -5841> 2014-11-08 18:15:01.276741 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276741, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -5840> 2014-11-08 18:15:01.276745 7fc5b9b29700 5 -- op tracker -- seq: 47049, time: 2014-11-08 18:15:01.276744, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -5839> 2014-11-08 18:15:01.276750 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2250 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x234e5640 con 0x1e50aec0 -5838> 2014-11-08 18:15:01.276759 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276626, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -5837> 2014-11-08 18:15:01.276763 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276627, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -5836> 2014-11-08 18:15:01.276767 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276645, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -5835> 2014-11-08 18:15:01.276770 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276758, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -5834> 2014-11-08 18:15:01.276772 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276772, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -5833> 2014-11-08 18:15:01.276776 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276776, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -5832> 2014-11-08 18:15:01.276781 7fc5b9b29700 5 -- op tracker -- seq: 47050, time: 2014-11-08 18:15:01.276780, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -5831> 2014-11-08 18:15:01.340075 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2251 ==== MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (654521925 0 0) 0x24840c00 con 0x1e50aec0 -5830> 2014-11-08 18:15:01.340136 7fc5a56ef700 5 -- op tracker -- seq: 47051, time: 2014-11-08 18:15:01.301017, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5829> 2014-11-08 18:15:01.340154 7fc5a56ef700 5 -- op tracker -- seq: 47051, time: 2014-11-08 18:15:01.301022, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5828> 2014-11-08 18:15:01.340167 7fc5a56ef700 5 -- op tracker -- seq: 47051, time: 2014-11-08 18:15:01.340064, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5827> 2014-11-08 18:15:01.340179 7fc5a56ef700 5 -- op tracker -- seq: 47051, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5826> 2014-11-08 18:15:01.340226 7fc5af314700 5 -- op tracker -- seq: 47051, time: 2014-11-08 18:15:01.340225, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5825> 2014-11-08 18:15:01.347840 7fc5a56ef700 5 -- op tracker -- seq: 47051, time: 2014-11-08 18:15:01.347840, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(e36f90b7/10000019aff.000000d7/head//4, version: 397'1045, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e36f90b7/10000019aff.000000d7/head//4@397'1045, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5824> 2014-11-08 18:15:01.399625 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1db1fbc0 con 0x1e50aec0 -5823> 2014-11-08 18:15:01.399780 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(e36f90b7/10000019aff.000000d7/head//4)]) v2 -- ?+0 0x23938400 con 0x1e50aec0 -5822> 2014-11-08 18:15:01.521996 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8549 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) v1 ==== 10488573+0+0 (185361358 0 0) 0x21a48000 con 0x1c5c0580 -5821> 2014-11-08 18:15:01.522037 7fc5a152e700 5 -- op tracker -- seq: 47052, time: 2014-11-08 18:15:01.282745, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) -5820> 2014-11-08 18:15:01.522052 7fc5a152e700 5 -- op tracker -- seq: 47052, time: 2014-11-08 18:15:01.282747, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) -5819> 2014-11-08 18:15:01.522115 7fc5a152e700 5 -- op tracker -- seq: 47052, time: 2014-11-08 18:15:01.521969, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) -5818> 2014-11-08 18:15:01.522125 7fc5a152e700 5 -- op tracker -- seq: 47052, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) -5817> 2014-11-08 18:15:01.522168 7fc5aeb13700 5 -- op tracker -- seq: 47052, time: 2014-11-08 18:15:01.522168, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) -5816> 2014-11-08 18:15:01.541678 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2252 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) v1 ==== 4195327+0+0 (3152252055 0 0) 0x224bc380 con 0x1e50aec0 -5815> 2014-11-08 18:15:01.541740 7fc5a56ef700 5 -- op tracker -- seq: 47053, time: 2014-11-08 18:15:01.380515, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) -5814> 2014-11-08 18:15:01.541754 7fc5a56ef700 5 -- op tracker -- seq: 47053, time: 2014-11-08 18:15:01.380521, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) -5813> 2014-11-08 18:15:01.541764 7fc5a56ef700 5 -- op tracker -- seq: 47053, time: 2014-11-08 18:15:01.541667, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) -5812> 2014-11-08 18:15:01.541772 7fc5a56ef700 5 -- op tracker -- seq: 47053, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) -5811> 2014-11-08 18:15:01.541814 7fc5ac30e700 5 -- op tracker -- seq: 47053, time: 2014-11-08 18:15:01.541814, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) -5810> 2014-11-08 18:15:01.560739 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(bd4edd13/1000002bc0a.0000001c/head//4, version: 3478'7256, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bd4edd13/1000002bc0a.0000001c/head//4@3478'7256, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7e7edd13/1000002b5ea.000004c4/head//4, version: 3473'6459, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7e7edd13/1000002b5ea.000004c4/head//4@3473'6459, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d859400 con 0x1c5c0580 -5809> 2014-11-08 18:15:01.568530 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(bd4edd13/1000002bc0a.0000001c/head//4, version: 3478'7256, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bd4edd13/1000002bc0a.0000001c/head//4@3478'7256, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7e7edd13/1000002b5ea.000004c4/head//4, version: 3473'6459, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7e7edd13/1000002b5ea.000004c4/head//4@3473'6459, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d858a00 con 0x1e50aec0 -5808> 2014-11-08 18:15:01.576473 7fc5ac30e700 5 -- op tracker -- seq: 47053, time: 2014-11-08 18:15:01.576472, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4419, attrs_read=2)) -5807> 2014-11-08 18:15:01.595630 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(44260518/1000002ad5b.00000044/head//4, version: 3468'6125, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(44260518/1000002ad5b.00000044/head//4@3468'6125, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(75f60518/1000001a0eb.00000616/head//4, version: 397'1852, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75f60518/1000001a0eb.00000616/head//4@397'1852, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cac70518/10000016074.00000000/head//4, version: 397'677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(cac70518/10000016074.00000000/head//4@397'677, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9e080518/1000001a2e2.00000011/head//4, version: 397'2025, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9e080518/1000001a2e2.00000011/head//4@ 397'2025, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(22c80518/1000002b9f1.0000000d/head//4, version: 3473'7268, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22c80518/1000002b9f1.0000000d/head//4@3473'7268, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x21617200 con 0x1e50aec0 -5806> 2014-11-08 18:15:01.595943 7fc5aeb13700 5 -- op tracker -- seq: 47052, time: 2014-11-08 18:15:01.595942, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4418, attrs_read=5)) -5805> 2014-11-08 18:15:01.673232 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1504 ==== pg_stats_ack(4 pgs tid 1329) v1 ==== 120+0+0 (641074115 0 0) 0x237181e0 con 0x44ed6a0 -5804> 2014-11-08 18:15:01.676207 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8550 ==== MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (3597629535 0 0) 0x229b9200 con 0x1c5c0580 -5803> 2014-11-08 18:15:01.676250 7fc5a152e700 5 -- op tracker -- seq: 47054, time: 2014-11-08 18:15:01.535626, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5802> 2014-11-08 18:15:01.676270 7fc5a152e700 5 -- op tracker -- seq: 47054, time: 2014-11-08 18:15:01.535627, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5801> 2014-11-08 18:15:01.676284 7fc5a152e700 5 -- op tracker -- seq: 47054, time: 2014-11-08 18:15:01.676197, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5800> 2014-11-08 18:15:01.676298 7fc5a152e700 5 -- op tracker -- seq: 47054, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5799> 2014-11-08 18:15:01.676345 7fc5afb15700 5 -- op tracker -- seq: 47054, time: 2014-11-08 18:15:01.676345, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5798> 2014-11-08 18:15:01.676475 7fc5afb15700 5 -- op tracker -- seq: 47054, time: 2014-11-08 18:15:01.676475, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(c5b07b16/1000001b50d.000002e2/head//4, version: 3033'3234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5b07b16/1000001b50d.000002e2/head//4@3033'3234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5da17b16/10000019cda.00000046/head//4, version: 397'1311, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5da17b16/10000019cda.00000046/head//4@397'1311, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(38337b16/1000001a12a.00000227/head//4, version: 397'1717, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38337b16/1000001a12a.00000227/head//4@397'1717, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5797> 2014-11-08 18:15:01.678857 7fc5c362f700 5 osd.9 5773 tick -5796> 2014-11-08 18:15:01.722538 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8551 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1decc560 con 0x1c5c0580 -5795> 2014-11-08 18:15:01.722578 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722423, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -5794> 2014-11-08 18:15:01.722587 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722428, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -5793> 2014-11-08 18:15:01.722589 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722508, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -5792> 2014-11-08 18:15:01.722592 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722573, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -5791> 2014-11-08 18:15:01.722595 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722595, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -5790> 2014-11-08 18:15:01.722605 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722605, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -5789> 2014-11-08 18:15:01.722610 7fc5b9b29700 5 -- op tracker -- seq: 47055, time: 2014-11-08 18:15:01.722610, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -5788> 2014-11-08 18:15:01.722763 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8552 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1deccce0 con 0x1c5c0580 -5787> 2014-11-08 18:15:01.722772 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722701, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -5786> 2014-11-08 18:15:01.722776 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722702, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -5785> 2014-11-08 18:15:01.722779 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722753, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -5784> 2014-11-08 18:15:01.722781 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722771, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -5783> 2014-11-08 18:15:01.722784 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722784, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -5782> 2014-11-08 18:15:01.722787 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722787, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -5781> 2014-11-08 18:15:01.722791 7fc5b9b29700 5 -- op tracker -- seq: 47056, time: 2014-11-08 18:15:01.722791, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -5780> 2014-11-08 18:15:01.722837 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8553 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) v2 ==== 175+0+0 (3389460222 0 0) 0x1d859400 con 0x1c5c0580 -5779> 2014-11-08 18:15:01.722856 7fc5a152e700 5 -- op tracker -- seq: 47057, time: 2014-11-08 18:15:01.722807, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -5778> 2014-11-08 18:15:01.722866 7fc5a152e700 5 -- op tracker -- seq: 47057, time: 2014-11-08 18:15:01.722808, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -5777> 2014-11-08 18:15:01.722876 7fc5a152e700 5 -- op tracker -- seq: 47057, time: 2014-11-08 18:15:01.722836, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -5776> 2014-11-08 18:15:01.722887 7fc5a152e700 5 -- op tracker -- seq: 47057, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -5775> 2014-11-08 18:15:01.722929 7fc5ac30e700 5 -- op tracker -- seq: 47057, time: 2014-11-08 18:15:01.722928, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -5774> 2014-11-08 18:15:01.722973 7fc5a152e700 5 -- op tracker -- seq: 47057, time: 2014-11-08 18:15:01.722973, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -5773> 2014-11-08 18:15:01.761215 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2253 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x1db1fbc0 con 0x1e50aec0 -5772> 2014-11-08 18:15:01.761250 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761055, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -5771> 2014-11-08 18:15:01.761260 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761058, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -5770> 2014-11-08 18:15:01.761264 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761193, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -5769> 2014-11-08 18:15:01.761269 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761245, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -5768> 2014-11-08 18:15:01.761274 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761274, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -5767> 2014-11-08 18:15:01.761283 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761283, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -5766> 2014-11-08 18:15:01.761291 7fc5b9b29700 5 -- op tracker -- seq: 47058, time: 2014-11-08 18:15:01.761291, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -5765> 2014-11-08 18:15:01.761302 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2254 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x2285d460 con 0x1e50aec0 -5764> 2014-11-08 18:15:01.761311 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761214, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -5763> 2014-11-08 18:15:01.761315 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761215, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -5762> 2014-11-08 18:15:01.761319 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761234, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -5761> 2014-11-08 18:15:01.761323 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761310, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -5760> 2014-11-08 18:15:01.761328 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761328, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -5759> 2014-11-08 18:15:01.761333 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761333, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -5758> 2014-11-08 18:15:01.761338 7fc5b9b29700 5 -- op tracker -- seq: 47059, time: 2014-11-08 18:15:01.761338, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -5757> 2014-11-08 18:15:01.809704 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1db185a0 con 0x1c5c0580 -5756> 2014-11-08 18:15:01.809820 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2583ad00 con 0x1c5c0580 -5755> 2014-11-08 18:15:01.809831 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25ed01e0 con 0x1c5c0580 -5754> 2014-11-08 18:15:01.809840 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(c5b07b16/1000001b50d.000002e2/head//4),PushReplyOp(5da17b16/10000019cda.00000046/head//4),PushReplyOp(38337b16/1000001a12a.00000227/head//4)]) v2 -- ?+0 0x24847400 con 0x1c5c0580 -5753> 2014-11-08 18:15:01.813625 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8554 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) v3 ==== 757+0+0 (2403829558 0 0) 0x22767080 con 0x1c5c0580 -5752> 2014-11-08 18:15:01.813656 7fc5a152e700 5 -- op tracker -- seq: 47060, time: 2014-11-08 18:15:01.813546, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5751> 2014-11-08 18:15:01.813670 7fc5a152e700 5 -- op tracker -- seq: 47060, time: 2014-11-08 18:15:01.813549, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5750> 2014-11-08 18:15:01.813680 7fc5a152e700 5 -- op tracker -- seq: 47060, time: 2014-11-08 18:15:01.813607, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5749> 2014-11-08 18:15:01.813696 7fc5a152e700 5 -- op tracker -- seq: 47060, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5748> 2014-11-08 18:15:01.813739 7fc5ad310700 5 -- op tracker -- seq: 47060, time: 2014-11-08 18:15:01.813739, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5747> 2014-11-08 18:15:01.813750 7fc5ad310700 5 -- op tracker -- seq: 47060, time: 2014-11-08 18:15:01.813750, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5746> 2014-11-08 18:15:01.813797 7fc5ad310700 5 -- op tracker -- seq: 47060, time: 2014-11-08 18:15:01.813797, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 38337b16/1000001a12a.00000227/head//4) -5745> 2014-11-08 18:15:01.881046 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2255 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1c835640 con 0x1e50aec0 -5744> 2014-11-08 18:15:01.881088 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.880954, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5743> 2014-11-08 18:15:01.881098 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.880958, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5742> 2014-11-08 18:15:01.881103 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.881024, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5741> 2014-11-08 18:15:01.881107 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.881082, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5740> 2014-11-08 18:15:01.881112 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.881112, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5739> 2014-11-08 18:15:01.881122 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.881122, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5738> 2014-11-08 18:15:01.881131 7fc5b9b29700 5 -- op tracker -- seq: 47061, time: 2014-11-08 18:15:01.881131, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5737> 2014-11-08 18:15:01.881142 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2256 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x1d5e1860 con 0x1e50aec0 -5736> 2014-11-08 18:15:01.881151 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881046, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5735> 2014-11-08 18:15:01.881156 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881046, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5734> 2014-11-08 18:15:01.881160 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881065, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5733> 2014-11-08 18:15:01.881165 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881150, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5732> 2014-11-08 18:15:01.881168 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881168, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5731> 2014-11-08 18:15:01.881171 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881171, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5730> 2014-11-08 18:15:01.881174 7fc5b9b29700 5 -- op tracker -- seq: 47062, time: 2014-11-08 18:15:01.881174, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5729> 2014-11-08 18:15:01.881180 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2257 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x224523a0 con 0x1e50aec0 -5728> 2014-11-08 18:15:01.881186 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881070, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5727> 2014-11-08 18:15:01.881189 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881071, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5726> 2014-11-08 18:15:01.881194 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881097, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5725> 2014-11-08 18:15:01.881198 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881185, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5724> 2014-11-08 18:15:01.881202 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881202, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5723> 2014-11-08 18:15:01.881208 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881208, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5722> 2014-11-08 18:15:01.881213 7fc5b9b29700 5 -- op tracker -- seq: 47063, time: 2014-11-08 18:15:01.881213, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5721> 2014-11-08 18:15:01.881220 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2258 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22456720 con 0x1e50aec0 -5720> 2014-11-08 18:15:01.881236 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881115, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5719> 2014-11-08 18:15:01.881243 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881116, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5718> 2014-11-08 18:15:01.881246 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881146, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5717> 2014-11-08 18:15:01.881248 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881235, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5716> 2014-11-08 18:15:01.881251 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881251, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5715> 2014-11-08 18:15:01.881255 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881255, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5714> 2014-11-08 18:15:01.881259 7fc5b9b29700 5 -- op tracker -- seq: 47064, time: 2014-11-08 18:15:01.881259, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5713> 2014-11-08 18:15:01.881266 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2259 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22451680 con 0x1e50aec0 -5712> 2014-11-08 18:15:01.881274 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881155, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -5711> 2014-11-08 18:15:01.881279 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881159, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -5710> 2014-11-08 18:15:01.881282 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881227, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -5709> 2014-11-08 18:15:01.881284 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881273, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -5708> 2014-11-08 18:15:01.881287 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881287, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -5707> 2014-11-08 18:15:01.881290 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881290, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -5706> 2014-11-08 18:15:01.881293 7fc5b9b29700 5 -- op tracker -- seq: 47065, time: 2014-11-08 18:15:01.881293, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -5705> 2014-11-08 18:15:02.047765 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5704> 2014-11-08 18:15:02.047795 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1e536ea0 con 0x1cae2940 -5703> 2014-11-08 18:15:02.047817 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1e894ec0 con 0x1cae2680 -5702> 2014-11-08 18:15:02.047826 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x246ca760 con 0x1c5c02c0 -5701> 2014-11-08 18:15:02.047835 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x257c6ea0 con 0x1cae0160 -5700> 2014-11-08 18:15:02.047848 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1d324560 con 0x1cae0f20 -5699> 2014-11-08 18:15:02.047858 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1d4f2760 con 0x1cae0c60 -5698> 2014-11-08 18:15:02.047869 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x23d76360 con 0x1cae23c0 -5697> 2014-11-08 18:15:02.047879 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x22e981e0 con 0x1cae56a0 -5696> 2014-11-08 18:15:02.047889 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x2583eea0 con 0x1cbf9ce0 -5695> 2014-11-08 18:15:02.047899 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1f4c30c0 con 0x1d603020 -5694> 2014-11-08 18:15:02.047909 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x2583a760 con 0x1dbd9b80 -5693> 2014-11-08 18:15:02.047918 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x24880000 con 0x1d235d80 -5692> 2014-11-08 18:15:02.047927 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x246cf620 con 0x1cae1fa0 -5691> 2014-11-08 18:15:02.047936 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x24882580 con 0x1cae1ce0 -5690> 2014-11-08 18:15:02.047945 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x227b6360 con 0x1cae1340 -5689> 2014-11-08 18:15:02.047955 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1dbfd460 con 0x1cae1080 -5688> 2014-11-08 18:15:02.047964 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x22e9f800 con 0x1cae1a20 -5687> 2014-11-08 18:15:02.047973 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.047792) v2 -- ?+0 0x1d6323a0 con 0x1cae2c00 -5686> 2014-11-08 18:15:02.048592 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x259f1860 con 0x1cae1340 -5685> 2014-11-08 18:15:02.048613 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1d046360 con 0x1cae1080 -5684> 2014-11-08 18:15:02.048622 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x23686360 con 0x1cae2940 -5683> 2014-11-08 18:15:02.048632 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x25ed1c20 con 0x1cae56a0 -5682> 2014-11-08 18:15:02.048642 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x257c0780 con 0x1c5c02c0 -5681> 2014-11-08 18:15:02.048652 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x2290da00 con 0x1cae0160 -5680> 2014-11-08 18:15:02.048665 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1e6a8b40 con 0x1cae0f20 -5679> 2014-11-08 18:15:02.048675 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x2127c380 con 0x1cae2c00 -5678> 2014-11-08 18:15:02.048820 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x2495eae0 con 0x1cae23c0 -5677> 2014-11-08 18:15:02.048835 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 506 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x2234f260 con 0x1d603020 -5676> 2014-11-08 18:15:02.048845 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 313 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1caaf080 con 0x1d235d80 -5675> 2014-11-08 18:15:02.048858 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1e896720 con 0x1cae0c60 -5674> 2014-11-08 18:15:02.048868 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1eac92c0 con 0x1cae1fa0 -5673> 2014-11-08 18:15:02.048897 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x4537440 con 0x1cae1ce0 -5672> 2014-11-08 18:15:02.048908 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1d1b2760 con 0x1cae1a20 -5671> 2014-11-08 18:15:02.048916 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3726 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x1d5e14a0 con 0x1cae2680 -5670> 2014-11-08 18:15:02.048926 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 313 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x2583aee0 con 0x1dbd9b80 -5669> 2014-11-08 18:15:02.048960 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 506 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.047792) v2 ==== 47+0+0 (2737028137 0 0) 0x246c90e0 con 0x1cbf9ce0 -5668> 2014-11-08 18:15:02.336624 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8555 ==== MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (2910708121 0 0) 0x24847400 con 0x1c5c0580 -5667> 2014-11-08 18:15:02.336680 7fc5a152e700 5 -- op tracker -- seq: 47066, time: 2014-11-08 18:15:02.258411, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5666> 2014-11-08 18:15:02.336753 7fc5a152e700 5 -- op tracker -- seq: 47066, time: 2014-11-08 18:15:02.258421, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5665> 2014-11-08 18:15:02.336772 7fc5a152e700 5 -- op tracker -- seq: 47066, time: 2014-11-08 18:15:02.336589, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5664> 2014-11-08 18:15:02.336791 7fc5a152e700 5 -- op tracker -- seq: 47066, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5663> 2014-11-08 18:15:02.336845 7fc5afb15700 5 -- op tracker -- seq: 47066, time: 2014-11-08 18:15:02.336845, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5662> 2014-11-08 18:15:02.336979 7fc5afb15700 5 -- op tracker -- seq: 47066, time: 2014-11-08 18:15:02.336979, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(d1c37b16/1000002a2a2.00000022/head//4, version: 3033'4785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d1c37b16/1000002a2a2.00000022/head//4@3033'4785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(57d37b16/1000002b624.00000050/head//4, version: 3473'6742, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(57d37b16/1000002b624.00000050/head//4@3473'6742, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1cd37b16/1000002a350.00000004/head//4, version: 3033'4823, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1cd37b16/1000002a350.00000004/head//4@3033'4823, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5661> 2014-11-08 18:15:02.463245 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25ed4ce0 con 0x1c5c0580 -5660> 2014-11-08 18:15:02.463289 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d06eae0 con 0x1c5c0580 -5659> 2014-11-08 18:15:02.463393 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d06a1c0 con 0x1c5c0580 -5658> 2014-11-08 18:15:02.463403 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(d1c37b16/1000002a2a2.00000022/head//4),PushReplyOp(57d37b16/1000002b624.00000050/head//4),PushReplyOp(1cd37b16/1000002a350.00000004/head//4)]) v2 -- ?+0 0x229b9200 con 0x1c5c0580 -5657> 2014-11-08 18:15:02.467442 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8556 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) v3 ==== 757+0+0 (1927208421 0 0) 0x2482da00 con 0x1c5c0580 -5656> 2014-11-08 18:15:02.467491 7fc5a152e700 5 -- op tracker -- seq: 47067, time: 2014-11-08 18:15:02.467305, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5655> 2014-11-08 18:15:02.467505 7fc5a152e700 5 -- op tracker -- seq: 47067, time: 2014-11-08 18:15:02.467309, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5654> 2014-11-08 18:15:02.467516 7fc5a152e700 5 -- op tracker -- seq: 47067, time: 2014-11-08 18:15:02.467414, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5653> 2014-11-08 18:15:02.467525 7fc5a152e700 5 -- op tracker -- seq: 47067, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5652> 2014-11-08 18:15:02.467579 7fc5ad310700 5 -- op tracker -- seq: 47067, time: 2014-11-08 18:15:02.467579, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5651> 2014-11-08 18:15:02.467592 7fc5ad310700 5 -- op tracker -- seq: 47067, time: 2014-11-08 18:15:02.467592, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5650> 2014-11-08 18:15:02.467644 7fc5ad310700 5 -- op tracker -- seq: 47067, time: 2014-11-08 18:15:02.467644, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 1cd37b16/1000002a350.00000004/head//4) -5649> 2014-11-08 18:15:02.548371 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2011 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5648> 2014-11-08 18:15:02.548404 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x23d72940 con 0x1cae2940 -5647> 2014-11-08 18:15:02.548425 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1e8905a0 con 0x1cae2680 -5646> 2014-11-08 18:15:02.548435 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x223483c0 con 0x1c5c02c0 -5645> 2014-11-08 18:15:02.548444 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1dece900 con 0x1cae0160 -5644> 2014-11-08 18:15:02.548454 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x224579e0 con 0x1cae0f20 -5643> 2014-11-08 18:15:02.548463 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1f4c1680 con 0x1cae0c60 -5642> 2014-11-08 18:15:02.548472 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1d4f1c20 con 0x1cae23c0 -5641> 2014-11-08 18:15:02.548481 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1dbfc380 con 0x1cae56a0 -5640> 2014-11-08 18:15:02.548490 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x229d4380 con 0x1cbf9ce0 -5639> 2014-11-08 18:15:02.548499 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x2515aee0 con 0x1d603020 -5638> 2014-11-08 18:15:02.548508 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x246cd820 con 0x1dbd9b80 -5637> 2014-11-08 18:15:02.548518 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x227b6720 con 0x1d235d80 -5636> 2014-11-08 18:15:02.548527 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1e891a40 con 0x1cae1fa0 -5635> 2014-11-08 18:15:02.548535 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x22e990e0 con 0x1cae1ce0 -5634> 2014-11-08 18:15:02.548589 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x24887440 con 0x1cae1340 -5633> 2014-11-08 18:15:02.548598 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x1db1c740 con 0x1cae1080 -5632> 2014-11-08 18:15:02.548607 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x23d75460 con 0x1cae1a20 -5631> 2014-11-08 18:15:02.548615 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:02.548400) v2 -- ?+0 0x229d0960 con 0x1cae2c00 -5630> 2014-11-08 18:15:02.549090 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x25ed2d00 con 0x1cae56a0 -5629> 2014-11-08 18:15:02.549170 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x23680960 con 0x1cae2940 -5628> 2014-11-08 18:15:02.549392 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x2495a1c0 con 0x1cae23c0 -5627> 2014-11-08 18:15:02.549407 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1d5e2940 con 0x1cae2680 -5626> 2014-11-08 18:15:02.549428 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1e531a40 con 0x1cae0c60 -5625> 2014-11-08 18:15:02.549451 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1e6ab0c0 con 0x1cae0f20 -5624> 2014-11-08 18:15:02.549466 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 507 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x22348f00 con 0x1d603020 -5623> 2014-11-08 18:15:02.549544 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x45330c0 con 0x1cae1ce0 -5622> 2014-11-08 18:15:02.549562 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 314 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x25838d20 con 0x1dbd9b80 -5621> 2014-11-08 18:15:02.549595 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 314 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1d637bc0 con 0x1d235d80 -5620> 2014-11-08 18:15:02.549605 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x257c0b40 con 0x1c5c02c0 -5619> 2014-11-08 18:15:02.549614 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x2290c740 con 0x1cae0160 -5618> 2014-11-08 18:15:02.549622 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 507 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1d634740 con 0x1cbf9ce0 -5617> 2014-11-08 18:15:02.549651 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1eac8d20 con 0x1cae1fa0 -5616> 2014-11-08 18:15:02.549784 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x259f7800 con 0x1cae1340 -5615> 2014-11-08 18:15:02.549951 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1d047bc0 con 0x1cae1080 -5614> 2014-11-08 18:15:02.549966 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x2127c1a0 con 0x1cae2c00 -5613> 2014-11-08 18:15:02.549985 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3727 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:02.548400) v2 ==== 47+0+0 (370010559 0 0) 0x1d1b6900 con 0x1cae1a20 -5612> 2014-11-08 18:15:02.679029 7fc5c362f700 5 osd.9 5773 tick -5611> 2014-11-08 18:15:02.716731 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8557 ==== MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3136357579 0 0) 0x229b9200 con 0x1c5c0580 -5610> 2014-11-08 18:15:02.716768 7fc5a152e700 5 -- op tracker -- seq: 47068, time: 2014-11-08 18:15:02.680999, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5609> 2014-11-08 18:15:02.716780 7fc5a152e700 5 -- op tracker -- seq: 47068, time: 2014-11-08 18:15:02.681001, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5608> 2014-11-08 18:15:02.716788 7fc5a152e700 5 -- op tracker -- seq: 47068, time: 2014-11-08 18:15:02.716720, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5607> 2014-11-08 18:15:02.716795 7fc5a152e700 5 -- op tracker -- seq: 47068, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5606> 2014-11-08 18:15:02.716837 7fc5afb15700 5 -- op tracker -- seq: 47068, time: 2014-11-08 18:15:02.716837, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5605> 2014-11-08 18:15:02.716923 7fc5afb15700 5 -- op tracker -- seq: 47068, time: 2014-11-08 18:15:02.716923, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(84047b16/1000001a1dd.000001f9/head//4, version: 397'1786, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(84047b16/1000001a1dd.000001f9/head//4@397'1786, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5604> 2014-11-08 18:15:02.826217 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d06bfc0 con 0x1c5c0580 -5603> 2014-11-08 18:15:03.047934 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2260 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) v2 ==== 240+0+0 (1391583129 0 0) 0x21617200 con 0x1e50aec0 -5602> 2014-11-08 18:15:03.047973 7fc5a56ef700 5 -- op tracker -- seq: 47069, time: 2014-11-08 18:15:03.047832, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) -5601> 2014-11-08 18:15:03.047983 7fc5a56ef700 5 -- op tracker -- seq: 47069, time: 2014-11-08 18:15:03.047837, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) -5600> 2014-11-08 18:15:03.047987 7fc5a56ef700 5 -- op tracker -- seq: 47069, time: 2014-11-08 18:15:03.047925, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) -5599> 2014-11-08 18:15:03.047991 7fc5a56ef700 5 -- op tracker -- seq: 47069, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) -5598> 2014-11-08 18:15:03.048040 7fc5ad310700 5 -- op tracker -- seq: 47069, time: 2014-11-08 18:15:03.048040, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) -5597> 2014-11-08 18:15:03.048147 7fc5ad310700 5 -- op tracker -- seq: 47069, time: 2014-11-08 18:15:03.048146, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1bd0e484/1000002aaa9.00000053/head//4),PushReplyOp(f2f0e484/1000001b4fe.00000089/head//4),PushReplyOp(7572e484/10000019cfb.00000321/head//4)]) -5596> 2014-11-08 18:15:03.051411 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3710 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:03.049000) v2 ==== 47+0+0 (1594170676 0 0) 0x1ec24560 con 0x1c9f8580 -5595> 2014-11-08 18:15:03.051437 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.049000) v2 -- ?+0 0x2336f800 con 0x1c9f8580 -5594> 2014-11-08 18:15:03.051749 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3710 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:03.049000) v2 ==== 47+0+0 (1594170676 0 0) 0x227ac740 con 0x1c8b7d20 -5593> 2014-11-08 18:15:03.051763 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.049000) v2 -- ?+0 0x239ea3a0 con 0x1c8b7d20 -5592> 2014-11-08 18:15:03.306874 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3679 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:03.304490) v2 ==== 47+0+0 (2047310825 0 0) 0x23145fa0 con 0x1cce3440 -5591> 2014-11-08 18:15:03.306901 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.304490) v2 -- ?+0 0x227ac740 con 0x1cce3440 -5590> 2014-11-08 18:15:03.306936 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3679 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:03.304490) v2 ==== 47+0+0 (2047310825 0 0) 0x224a7bc0 con 0x1cce2ec0 -5589> 2014-11-08 18:15:03.306948 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.304490) v2 -- ?+0 0x1ec24560 con 0x1cce2ec0 -5588> 2014-11-08 18:15:03.403005 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 500 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:03.400600) v2 ==== 47+0+0 (3127269949 0 0) 0x1e6af800 con 0x1eb9c8e0 -5587> 2014-11-08 18:15:03.403030 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.400600) v2 -- ?+0 0x23145fa0 con 0x1eb9c8e0 -5586> 2014-11-08 18:15:03.403196 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 500 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:03.400600) v2 ==== 47+0+0 (3127269949 0 0) 0x1e6adbe0 con 0x1eb9d800 -5585> 2014-11-08 18:15:03.403208 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.400600) v2 -- ?+0 0x224a7bc0 con 0x1eb9d800 -5584> 2014-11-08 18:15:03.649011 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5583> 2014-11-08 18:15:03.649046 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1d4f2d00 con 0x1cae2940 -5582> 2014-11-08 18:15:03.649070 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1dbfe360 con 0x1cae2680 -5581> 2014-11-08 18:15:03.649082 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1d6341a0 con 0x1c5c02c0 -5580> 2014-11-08 18:15:03.649091 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1d634560 con 0x1cae0160 -5579> 2014-11-08 18:15:03.649100 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x2515b0c0 con 0x1cae0f20 -5578> 2014-11-08 18:15:03.649110 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x2583cec0 con 0x1cae0c60 -5577> 2014-11-08 18:15:03.649119 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1decda00 con 0x1cae23c0 -5576> 2014-11-08 18:15:03.649129 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x22454380 con 0x1cae56a0 -5575> 2014-11-08 18:15:03.649138 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x223492c0 con 0x1cbf9ce0 -5574> 2014-11-08 18:15:03.649147 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1dbffda0 con 0x1d603020 -5573> 2014-11-08 18:15:03.649161 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x22450780 con 0x1dbd9b80 -5572> 2014-11-08 18:15:03.649170 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x22e9dfa0 con 0x1d235d80 -5571> 2014-11-08 18:15:03.649180 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1eacf9e0 con 0x1cae1fa0 -5570> 2014-11-08 18:15:03.649189 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1d634ec0 con 0x1cae1ce0 -5569> 2014-11-08 18:15:03.649198 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x227b1c20 con 0x1cae1340 -5568> 2014-11-08 18:15:03.649206 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x246c8d20 con 0x1cae1080 -5567> 2014-11-08 18:15:03.649216 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1d6310e0 con 0x1cae1a20 -5566> 2014-11-08 18:15:03.649225 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:03.649043) v2 -- ?+0 0x1d4f0b40 con 0x1cae2c00 -5565> 2014-11-08 18:15:03.649513 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1d5e32a0 con 0x1cae2680 -5564> 2014-11-08 18:15:03.649538 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x236812c0 con 0x1cae2940 -5563> 2014-11-08 18:15:03.650210 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x2495eea0 con 0x1cae23c0 -5562> 2014-11-08 18:15:03.650227 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x25ed5a00 con 0x1cae56a0 -5561> 2014-11-08 18:15:03.650237 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 508 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1d6332a0 con 0x1cbf9ce0 -5560> 2014-11-08 18:15:03.650246 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 508 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x259f0d20 con 0x1d603020 -5559> 2014-11-08 18:15:03.650255 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 315 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x2583f440 con 0x1dbd9b80 -5558> 2014-11-08 18:15:03.650264 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 315 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1d636ae0 con 0x1d235d80 -5557> 2014-11-08 18:15:03.650272 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x257c14a0 con 0x1c5c02c0 -5556> 2014-11-08 18:15:03.650281 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x229083c0 con 0x1cae0160 -5555> 2014-11-08 18:15:03.650289 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1e6a85a0 con 0x1cae0f20 -5554> 2014-11-08 18:15:03.650298 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1e533480 con 0x1cae0c60 -5553> 2014-11-08 18:15:03.650307 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x259f32a0 con 0x1cae1340 -5552> 2014-11-08 18:15:03.650316 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1d042ee0 con 0x1cae1080 -5551> 2014-11-08 18:15:03.650324 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1eacb840 con 0x1cae1fa0 -5550> 2014-11-08 18:15:03.650332 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x227e6900 con 0x1cae1ce0 -5549> 2014-11-08 18:15:03.650341 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x1d1b5460 con 0x1cae1a20 -5548> 2014-11-08 18:15:03.650349 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3728 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:03.649043) v2 ==== 47+0+0 (3242597937 0 0) 0x2127f080 con 0x1cae2c00 -5547> 2014-11-08 18:15:03.679128 7fc5c362f700 5 osd.9 5773 tick -5546> 2014-11-08 18:15:04.679269 7fc5c362f700 5 osd.9 5773 tick -5545> 2014-11-08 18:15:04.751956 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5544> 2014-11-08 18:15:04.751992 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1dbff9e0 con 0x1cae2940 -5543> 2014-11-08 18:15:04.752016 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x22349860 con 0x1cae2680 -5542> 2014-11-08 18:15:04.752027 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x23d70780 con 0x1c5c02c0 -5541> 2014-11-08 18:15:04.752040 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x234e2d00 con 0x1cae0160 -5540> 2014-11-08 18:15:04.752049 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x234e6360 con 0x1cae0f20 -5539> 2014-11-08 18:15:04.752059 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x234e6cc0 con 0x1cae0c60 -5538> 2014-11-08 18:15:04.752068 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1d4f10e0 con 0x1cae23c0 -5537> 2014-11-08 18:15:04.752079 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1d4f0000 con 0x1cae56a0 -5536> 2014-11-08 18:15:04.752089 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1db1f440 con 0x1cbf9ce0 -5535> 2014-11-08 18:15:04.752099 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x239ee720 con 0x1d603020 -5534> 2014-11-08 18:15:04.752109 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x227ae900 con 0x1dbd9b80 -5533> 2014-11-08 18:15:04.752118 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x239eaee0 con 0x1d235d80 -5532> 2014-11-08 18:15:04.752127 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1db1d640 con 0x1cae1fa0 -5531> 2014-11-08 18:15:04.752135 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1d4f21c0 con 0x1cae1ce0 -5530> 2014-11-08 18:15:04.752153 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1d4f0f00 con 0x1cae1340 -5529> 2014-11-08 18:15:04.752163 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x1d4f2b20 con 0x1cae1080 -5528> 2014-11-08 18:15:04.752172 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x202690e0 con 0x1cae1a20 -5527> 2014-11-08 18:15:04.752181 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:04.751988) v2 -- ?+0 0x20268780 con 0x1cae2c00 -5526> 2014-11-08 18:15:04.752394 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1d5e4ce0 con 0x1cae2680 -5525> 2014-11-08 18:15:04.752582 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x24959680 con 0x1cae23c0 -5524> 2014-11-08 18:15:04.752597 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6812/5031236 509 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x259f5820 con 0x1d603020 -5523> 2014-11-08 18:15:04.752607 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x227e2940 con 0x1cae1ce0 -5522> 2014-11-08 18:15:04.752696 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1e6aa760 con 0x1cae0f20 -5521> 2014-11-08 18:15:04.752708 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1eaccce0 con 0x1cae1fa0 -5520> 2014-11-08 18:15:04.752718 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x25ed2760 con 0x1cae56a0 -5519> 2014-11-08 18:15:04.752764 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1e5314a0 con 0x1cae0c60 -5518> 2014-11-08 18:15:04.752923 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x2290fda0 con 0x1cae0160 -5517> 2014-11-08 18:15:04.752938 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1c830f00 con 0x1c5c02c0 -5516> 2014-11-08 18:15:04.752980 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x23684560 con 0x1cae2940 -5515> 2014-11-08 18:15:04.752998 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 316 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x2583f080 con 0x1dbd9b80 -5514> 2014-11-08 18:15:04.753387 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.4 192.168.1.16:6811/5031236 509 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1d6305a0 con 0x1cbf9ce0 -5513> 2014-11-08 18:15:04.753402 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 316 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1d636ea0 con 0x1d235d80 -5512> 2014-11-08 18:15:04.753411 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1d1b5fa0 con 0x1cae1a20 -5511> 2014-11-08 18:15:04.753420 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x2127bde0 con 0x1cae2c00 -5510> 2014-11-08 18:15:04.753479 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x259f0000 con 0x1cae1340 -5509> 2014-11-08 18:15:04.753494 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3729 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:04.751988) v2 ==== 47+0+0 (174516450 0 0) 0x1d042580 con 0x1cae1080 -5508> 2014-11-08 18:15:04.943791 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12062 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) v1 ==== 377+0+0 (2520554698 0 0) 0x1ea7b840 con 0x1c5c2680 -5507> 2014-11-08 18:15:04.943826 7fc5a6cfe700 5 -- op tracker -- seq: 47070, time: 2014-11-08 18:15:04.943678, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) -5506> 2014-11-08 18:15:04.943839 7fc5a6cfe700 5 -- op tracker -- seq: 47070, time: 2014-11-08 18:15:04.943683, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) -5505> 2014-11-08 18:15:04.943846 7fc5a6cfe700 5 -- op tracker -- seq: 47070, time: 2014-11-08 18:15:04.943783, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) -5504> 2014-11-08 18:15:04.943853 7fc5a6cfe700 5 -- op tracker -- seq: 47070, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) -5503> 2014-11-08 18:15:04.943909 7fc5aeb13700 5 -- op tracker -- seq: 47070, time: 2014-11-08 18:15:04.943909, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) -5502> 2014-11-08 18:15:05.484782 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=31d2e484/1000002abe7.0000000b/head//4,6a43e484/100000297d3.0000005e/head//4,ecb3e484/1000001a2bb.00000088/head//4)) v1 -- ?+0 0x1ea7d7c0 con 0x1c5c0580 -5501> 2014-11-08 18:15:05.484826 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7a880 con 0x42c86e0 -5500> 2014-11-08 18:15:05.484839 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 1c44e484/1000002abd8.00000035/head//4 -5499> 2014-11-08 18:15:05.484864 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb 7572e484/10000019cfb.00000321/head//4) v3 -- ?+0 0x23cea880 con 0x1e50aec0 -5498> 2014-11-08 18:15:05.484891 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7a880 con 0x42c86e0 -5497> 2014-11-08 18:15:05.484907 7fc5b9328700 5 -- op tracker -- seq: 47071, time: 2014-11-08 18:15:05.484834, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) -5496> 2014-11-08 18:15:05.484916 7fc5b9328700 5 -- op tracker -- seq: 47071, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) -5495> 2014-11-08 18:15:05.484920 7fc5b9328700 5 -- op tracker -- seq: 47071, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) -5494> 2014-11-08 18:15:05.484923 7fc5b9328700 5 -- op tracker -- seq: 47071, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) -5493> 2014-11-08 18:15:05.484958 7fc5afb15700 5 -- op tracker -- seq: 47071, time: 2014-11-08 18:15:05.484958, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) -5492> 2014-11-08 18:15:05.620704 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3742 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:05.619017) v2 ==== 47+0+0 (3916771402 0 0) 0x23c32ee0 con 0x1cce3180 -5491> 2014-11-08 18:15:05.620740 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:05.619017) v2 -- ?+0 0x1e6adbe0 con 0x1cce3180 -5490> 2014-11-08 18:15:05.620765 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3742 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:05.619017) v2 ==== 47+0+0 (3916771402 0 0) 0x1e8923a0 con 0x1cce3700 -5489> 2014-11-08 18:15:05.620815 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:05.619017) v2 -- ?+0 0x1e6af800 con 0x1cce3700 -5488> 2014-11-08 18:15:05.679370 7fc5c362f700 5 osd.9 5773 tick -5487> 2014-11-08 18:15:05.679418 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -5486> 2014-11-08 18:15:05.679428 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(4 pgs tid 1330 v 0) v1 -- ?+0 0x248a8840 con 0x44ed6a0 -5485> 2014-11-08 18:15:05.920004 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) v1 -- ?+0 0x44a6e00 con 0x42c86e0 -5484> 2014-11-08 18:15:05.920077 7fc5afb15700 5 -- op tracker -- seq: 47071, time: 2014-11-08 18:15:05.920076, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4420, to_read={31d2e484/1000002abe7.0000000b/head//4=0,4194304,6a43e484/100000297d3.0000005e/head//4=0,4194304,ecb3e484/1000001a2bb.00000088/head//4=0,4194304}, attrs_to_read=)) -5483> 2014-11-08 18:15:05.920080 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x44a6e00 con 0x42c86e0 -5482> 2014-11-08 18:15:05.920108 7fc5b9328700 5 -- op tracker -- seq: 47072, time: 2014-11-08 18:15:05.920065, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) -5481> 2014-11-08 18:15:05.920114 7fc5b9328700 5 -- op tracker -- seq: 47072, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) -5480> 2014-11-08 18:15:05.920118 7fc5b9328700 5 -- op tracker -- seq: 47072, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) -5479> 2014-11-08 18:15:05.920121 7fc5b9328700 5 -- op tracker -- seq: 47072, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) -5478> 2014-11-08 18:15:05.920160 7fc5ad310700 5 -- op tracker -- seq: 47072, time: 2014-11-08 18:15:05.920160, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) -5477> 2014-11-08 18:15:05.920227 7fc5ad310700 5 -- op tracker -- seq: 47072, time: 2014-11-08 18:15:05.920227, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=0)) -5476> 2014-11-08 18:15:06.001777 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25192, attrs_read=0)) v1 -- ?+0 0x21a48000 con 0x1c5c2680 -5475> 2014-11-08 18:15:06.014827 7fc5aeb13700 5 -- op tracker -- seq: 47070, time: 2014-11-08 18:15:06.014825, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25192, to_read={27ce3227/1000002b39d.00000077/head//4=0,4194304,c59f3227/1000002ad85.0000008d/head//4=0,4194304,22af3227/1000002b694.00000a45/head//4=0,4194304,67704227/1000002acb2.0000002d/head//4=0,4194304}, attrs_to_read=)) -5474> 2014-11-08 18:15:06.284096 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1505 ==== pg_stats_ack(4 pgs tid 1330) v1 ==== 120+0+0 (576353204 0 0) 0x2371d280 con 0x44ed6a0 -5473> 2014-11-08 18:15:06.397218 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 320 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:06.394953) v2 ==== 47+0+0 (3836477844 0 0) 0x233683c0 con 0x1eb9d6a0 -5472> 2014-11-08 18:15:06.397250 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 320 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:06.394953) v2 ==== 47+0+0 (3836477844 0 0) 0x239e9fe0 con 0x1eb9c780 -5471> 2014-11-08 18:15:06.397269 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:06.394953) v2 -- ?+0 0x23c32ee0 con 0x1eb9d6a0 -5470> 2014-11-08 18:15:06.397472 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:06.394953) v2 -- ?+0 0x1e8923a0 con 0x1eb9c780 -5469> 2014-11-08 18:15:06.550504 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8558 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) v1 ==== 6292958+0+0 (1604081516 0 0) 0x21a49e00 con 0x1c5c0580 -5468> 2014-11-08 18:15:06.550545 7fc5a152e700 5 -- op tracker -- seq: 47073, time: 2014-11-08 18:15:06.485981, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) -5467> 2014-11-08 18:15:06.550560 7fc5a152e700 5 -- op tracker -- seq: 47073, time: 2014-11-08 18:15:06.485986, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) -5466> 2014-11-08 18:15:06.550568 7fc5a152e700 5 -- op tracker -- seq: 47073, time: 2014-11-08 18:15:06.550439, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) -5465> 2014-11-08 18:15:06.550577 7fc5a152e700 5 -- op tracker -- seq: 47073, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) -5464> 2014-11-08 18:15:06.550625 7fc5afb15700 5 -- op tracker -- seq: 47073, time: 2014-11-08 18:15:06.550625, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) -5463> 2014-11-08 18:15:06.563504 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(31d2e484/1000002abe7.0000000b/head//4, version: 3468'5863, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(31d2e484/1000002abe7.0000000b/head//4@3468'5863, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6a43e484/100000297d3.0000005e/head//4, version: 3033'4353, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6a43e484/100000297d3.0000005e/head//4@3033'4353, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ecb3e484/1000001a2bb.00000088/head//4, version: 397'1834, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ecb3e484/1000001a2bb.00000088/head//4@397'1834, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b9200 con 0x1e50aec0 -5462> 2014-11-08 18:15:06.563775 7fc5afb15700 5 -- op tracker -- seq: 47073, time: 2014-11-08 18:15:06.563775, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4420, attrs_read=3)) -5461> 2014-11-08 18:15:06.679585 7fc5c362f700 5 osd.9 5773 tick -5460> 2014-11-08 18:15:06.743676 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2261 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x22450960 con 0x1e50aec0 -5459> 2014-11-08 18:15:06.743723 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743568, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -5458> 2014-11-08 18:15:06.743732 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743571, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -5457> 2014-11-08 18:15:06.743735 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743650, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -5456> 2014-11-08 18:15:06.743739 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743717, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -5455> 2014-11-08 18:15:06.743743 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743743, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -5454> 2014-11-08 18:15:06.743753 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743753, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -5453> 2014-11-08 18:15:06.743760 7fc5b9b29700 5 -- op tracker -- seq: 47074, time: 2014-11-08 18:15:06.743760, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -5452> 2014-11-08 18:15:06.743826 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2262 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x25fd05a0 con 0x1e50aec0 -5451> 2014-11-08 18:15:06.743836 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743779, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -5450> 2014-11-08 18:15:06.743840 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743780, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -5449> 2014-11-08 18:15:06.743842 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743816, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -5448> 2014-11-08 18:15:06.743847 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743835, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -5447> 2014-11-08 18:15:06.743850 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743850, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -5446> 2014-11-08 18:15:06.743854 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743854, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -5445> 2014-11-08 18:15:06.743857 7fc5b9b29700 5 -- op tracker -- seq: 47075, time: 2014-11-08 18:15:06.743857, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -5444> 2014-11-08 18:15:06.743902 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2263 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x4536cc0 con 0x1e50aec0 -5443> 2014-11-08 18:15:06.743911 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743871, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -5442> 2014-11-08 18:15:06.743916 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743871, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -5441> 2014-11-08 18:15:06.743921 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743892, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -5440> 2014-11-08 18:15:06.743924 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743910, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -5439> 2014-11-08 18:15:06.743928 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743928, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -5438> 2014-11-08 18:15:06.743934 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743934, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -5437> 2014-11-08 18:15:06.743939 7fc5b9b29700 5 -- op tracker -- seq: 47076, time: 2014-11-08 18:15:06.743939, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -5436> 2014-11-08 18:15:07.052601 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5435> 2014-11-08 18:15:07.052634 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x2026c560 con 0x1cae2940 -5434> 2014-11-08 18:15:07.052660 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x2026fbc0 con 0x1cae2680 -5433> 2014-11-08 18:15:07.052671 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x22454560 con 0x1c5c02c0 -5432> 2014-11-08 18:15:07.052681 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x22456540 con 0x1cae0160 -5431> 2014-11-08 18:15:07.052705 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1d072b20 con 0x1cae0f20 -5430> 2014-11-08 18:15:07.052717 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1d044ec0 con 0x1cae0c60 -5429> 2014-11-08 18:15:07.052725 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x22452d00 con 0x1cae23c0 -5428> 2014-11-08 18:15:07.052735 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x22452940 con 0x1cae56a0 -5427> 2014-11-08 18:15:07.052743 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x22452760 con 0x1cbf9ce0 -5426> 2014-11-08 18:15:07.052753 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x22450d20 con 0x1d603020 -5425> 2014-11-08 18:15:07.052762 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1eae1e00 con 0x1dbd9b80 -5424> 2014-11-08 18:15:07.052771 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1ec20000 con 0x1d235d80 -5423> 2014-11-08 18:15:07.052781 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1db192c0 con 0x1cae1fa0 -5422> 2014-11-08 18:15:07.052790 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1eae5280 con 0x1cae1ce0 -5421> 2014-11-08 18:15:07.052799 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x1d4f1fe0 con 0x1cae1340 -5420> 2014-11-08 18:15:07.052809 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x21225280 con 0x1cae1080 -5419> 2014-11-08 18:15:07.052818 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x2495e540 con 0x1cae1a20 -5418> 2014-11-08 18:15:07.052827 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.052630) v2 -- ?+0 0x2495f080 con 0x1cae2c00 -5417> 2014-11-08 18:15:07.053361 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1e5323a0 con 0x1cae0c60 -5416> 2014-11-08 18:15:07.053381 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x227e7bc0 con 0x1cae1ce0 -5415> 2014-11-08 18:15:07.053392 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1e6ac380 con 0x1cae0f20 -5414> 2014-11-08 18:15:07.053472 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x2127d460 con 0x1cae2c00 -5413> 2014-11-08 18:15:07.053653 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x229912c0 con 0x1cae1340 -5412> 2014-11-08 18:15:07.053668 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1d040000 con 0x1cae1080 -5411> 2014-11-08 18:15:07.053677 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1eacd820 con 0x1cae1fa0 -5410> 2014-11-08 18:15:07.053694 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1d1b30c0 con 0x1cae1a20 -5409> 2014-11-08 18:15:07.053704 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x23682580 con 0x1cae2940 -5408> 2014-11-08 18:15:07.053713 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1d5e4740 con 0x1cae2680 -5407> 2014-11-08 18:15:07.053721 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1c8341a0 con 0x1c5c02c0 -5406> 2014-11-08 18:15:07.053730 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x22859c20 con 0x1cae0160 -5405> 2014-11-08 18:15:07.054326 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x25ed1860 con 0x1cae56a0 -5404> 2014-11-08 18:15:07.054342 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3730 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x2495f260 con 0x1cae23c0 -5403> 2014-11-08 18:15:07.054605 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 317 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x1d634920 con 0x1d235d80 -5402> 2014-11-08 18:15:07.054620 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 317 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.052630) v2 ==== 47+0+0 (416993285 0 0) 0x25838780 con 0x1dbd9b80 -5401> 2014-11-08 18:15:07.553317 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5400> 2014-11-08 18:15:07.553351 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x2495e360 con 0x1cae2940 -5399> 2014-11-08 18:15:07.553377 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x227e05a0 con 0x1cae2680 -5398> 2014-11-08 18:15:07.553390 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x22858b40 con 0x1c5c02c0 -5397> 2014-11-08 18:15:07.553399 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x213bfbc0 con 0x1cae0160 -5396> 2014-11-08 18:15:07.553409 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x1d5e1a40 con 0x1cae0f20 -5395> 2014-11-08 18:15:07.553418 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x25e84380 con 0x1cae0c60 -5394> 2014-11-08 18:15:07.553427 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x22e9b840 con 0x1cae23c0 -5393> 2014-11-08 18:15:07.553443 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x22e9eea0 con 0x1cae56a0 -5392> 2014-11-08 18:15:07.553453 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x213bb660 con 0x1cbf9ce0 -5391> 2014-11-08 18:15:07.553462 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x213bad00 con 0x1d603020 -5390> 2014-11-08 18:15:07.553471 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x213bc740 con 0x1dbd9b80 -5389> 2014-11-08 18:15:07.553510 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d723a0 con 0x1d235d80 -5388> 2014-11-08 18:15:07.553519 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d74ce0 con 0x1cae1fa0 -5387> 2014-11-08 18:15:07.553528 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d76720 con 0x1cae1ce0 -5386> 2014-11-08 18:15:07.553536 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d70960 con 0x1cae1340 -5385> 2014-11-08 18:15:07.553545 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d76cc0 con 0x1cae1080 -5384> 2014-11-08 18:15:07.553553 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d701e0 con 0x1cae1a20 -5383> 2014-11-08 18:15:07.553562 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:07.553347) v2 -- ?+0 0x23d76540 con 0x1cae2c00 -5382> 2014-11-08 18:15:07.554344 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1d045fa0 con 0x1cae1080 -5381> 2014-11-08 18:15:07.554364 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x22859a40 con 0x1cae0160 -5380> 2014-11-08 18:15:07.554462 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1d1b1fe0 con 0x1cae1a20 -5379> 2014-11-08 18:15:07.554478 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x22991c20 con 0x1cae1340 -5378> 2014-11-08 18:15:07.554512 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x2127e180 con 0x1cae2c00 -5377> 2014-11-08 18:15:07.554862 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1c837bc0 con 0x1c5c02c0 -5376> 2014-11-08 18:15:07.554879 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x23684380 con 0x1cae2940 -5375> 2014-11-08 18:15:07.554905 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1eac9c20 con 0x1cae1fa0 -5374> 2014-11-08 18:15:07.554917 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1e6ac1a0 con 0x1cae0f20 -5373> 2014-11-08 18:15:07.554927 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x227e23a0 con 0x1cae1ce0 -5372> 2014-11-08 18:15:07.554949 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1e535dc0 con 0x1cae0c60 -5371> 2014-11-08 18:15:07.554961 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x1d5e4b00 con 0x1cae2680 -5370> 2014-11-08 18:15:07.555031 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x2495b2a0 con 0x1cae23c0 -5369> 2014-11-08 18:15:07.555154 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3731 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x25ed32a0 con 0x1cae56a0 -5368> 2014-11-08 18:15:07.555168 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 318 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x259f1680 con 0x1d235d80 -5367> 2014-11-08 18:15:07.555531 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 318 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:07.553347) v2 ==== 47+0+0 (1474457530 0 0) 0x2583cce0 con 0x1dbd9b80 -5366> 2014-11-08 18:15:07.679705 7fc5c362f700 5 osd.9 5773 tick -5365> 2014-11-08 18:15:08.103574 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 501 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.101310) v2 ==== 47+0+0 (2661161631 0 0) 0x1e6ada00 con 0x1eb9d800 -5364> 2014-11-08 18:15:08.103601 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.101310) v2 -- ?+0 0x233683c0 con 0x1eb9d800 -5363> 2014-11-08 18:15:08.103624 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 501 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.101310) v2 ==== 47+0+0 (2661161631 0 0) 0x1e6af620 con 0x1eb9c8e0 -5362> 2014-11-08 18:15:08.103634 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.101310) v2 -- ?+0 0x239e9fe0 con 0x1eb9c8e0 -5361> 2014-11-08 18:15:08.351846 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3711 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.349691) v2 ==== 47+0+0 (2998705405 0 0) 0x1d1b4ec0 con 0x1c9f8580 -5360> 2014-11-08 18:15:08.351882 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.349691) v2 -- ?+0 0x1e6ada00 con 0x1c9f8580 -5359> 2014-11-08 18:15:08.352231 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3711 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.349691) v2 ==== 47+0+0 (2998705405 0 0) 0x227ac1a0 con 0x1c8b7d20 -5358> 2014-11-08 18:15:08.352244 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.349691) v2 -- ?+0 0x1e6af620 con 0x1c8b7d20 -5357> 2014-11-08 18:15:08.609451 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3680 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.605655) v2 ==== 47+0+0 (2856098735 0 0) 0x234e79e0 con 0x1cce3440 -5356> 2014-11-08 18:15:08.609489 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.605655) v2 -- ?+0 0x227ac1a0 con 0x1cce3440 -5355> 2014-11-08 18:15:08.610260 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3680 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.605655) v2 ==== 47+0+0 (2856098735 0 0) 0x224a5dc0 con 0x1cce2ec0 -5354> 2014-11-08 18:15:08.610275 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.605655) v2 -- ?+0 0x1d1b4ec0 con 0x1cce2ec0 -5353> 2014-11-08 18:15:08.653961 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5352> 2014-11-08 18:15:08.653995 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x1d1b6ae0 con 0x1cae2940 -5351> 2014-11-08 18:15:08.654021 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22e9b0c0 con 0x1cae2680 -5350> 2014-11-08 18:15:08.654032 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22e99680 con 0x1c5c02c0 -5349> 2014-11-08 18:15:08.654043 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x1d1b1c20 con 0x1cae0160 -5348> 2014-11-08 18:15:08.654052 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22d505a0 con 0x1cae0f20 -5347> 2014-11-08 18:15:08.654062 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x2257e900 con 0x1cae0c60 -5346> 2014-11-08 18:15:08.654072 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x24959860 con 0x1cae23c0 -5345> 2014-11-08 18:15:08.654081 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x2495f9e0 con 0x1cae56a0 -5344> 2014-11-08 18:15:08.654090 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x2286fbc0 con 0x1cbf9ce0 -5343> 2014-11-08 18:15:08.654100 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22453de0 con 0x1d603020 -5342> 2014-11-08 18:15:08.654110 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22d51e00 con 0x1dbd9b80 -5341> 2014-11-08 18:15:08.654127 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22d54b00 con 0x1d235d80 -5340> 2014-11-08 18:15:08.654137 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22d55280 con 0x1cae1fa0 -5339> 2014-11-08 18:15:08.654146 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22d57620 con 0x1cae1ce0 -5338> 2014-11-08 18:15:08.654155 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22455dc0 con 0x1cae1340 -5337> 2014-11-08 18:15:08.654164 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22450000 con 0x1cae1080 -5336> 2014-11-08 18:15:08.654173 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x22d57440 con 0x1cae1a20 -5335> 2014-11-08 18:15:08.654181 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:08.653991) v2 -- ?+0 0x1decc1a0 con 0x1cae2c00 -5334> 2014-11-08 18:15:08.654462 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1d0423a0 con 0x1cae1080 -5333> 2014-11-08 18:15:08.654891 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1d1b0000 con 0x1cae1a20 -5332> 2014-11-08 18:15:08.654908 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x2127cce0 con 0x1cae2c00 -5331> 2014-11-08 18:15:08.654918 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x23685dc0 con 0x1cae2940 -5330> 2014-11-08 18:15:08.654927 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1d5e2d00 con 0x1cae2680 -5329> 2014-11-08 18:15:08.654936 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x24959a40 con 0x1cae23c0 -5328> 2014-11-08 18:15:08.654946 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x25ed7440 con 0x1cae56a0 -5327> 2014-11-08 18:15:08.654954 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1c835280 con 0x1c5c02c0 -5326> 2014-11-08 18:15:08.654962 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x2285c560 con 0x1cae0160 -5325> 2014-11-08 18:15:08.654970 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1e6abfc0 con 0x1cae0f20 -5324> 2014-11-08 18:15:08.654982 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1e535640 con 0x1cae0c60 -5323> 2014-11-08 18:15:08.655111 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x1eacb480 con 0x1cae1fa0 -5322> 2014-11-08 18:15:08.655165 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x2485d820 con 0x1cae1ce0 -5321> 2014-11-08 18:15:08.655222 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 319 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x259f4740 con 0x1d235d80 -5320> 2014-11-08 18:15:08.655235 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 319 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x2583c1a0 con 0x1dbd9b80 -5319> 2014-11-08 18:15:08.655334 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3732 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.653991) v2 ==== 47+0+0 (566622196 0 0) 0x22994740 con 0x1cae1340 -5318> 2014-11-08 18:15:08.679819 7fc5c362f700 5 osd.9 5773 tick -5317> 2014-11-08 18:15:08.853251 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3712 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.850894) v2 ==== 47+0+0 (4229670186 0 0) 0x227af260 con 0x1c8b7d20 -5316> 2014-11-08 18:15:08.853284 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.850894) v2 -- ?+0 0x234e79e0 con 0x1c8b7d20 -5315> 2014-11-08 18:15:08.853392 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3712 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:08.850894) v2 ==== 47+0+0 (4229670186 0 0) 0x1d1b41a0 con 0x1c9f8580 -5314> 2014-11-08 18:15:08.853405 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:08.850894) v2 -- ?+0 0x224a5dc0 con 0x1c9f8580 -5313> 2014-11-08 18:15:09.115633 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(84047b16/1000001a1dd.000001f9/head//4)]) v2 -- ?+0 0x24847400 con 0x1c5c0580 -5312> 2014-11-08 18:15:09.118697 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8559 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) v3 ==== 757+0+0 (2311206253 0 0) 0x2366d580 con 0x1c5c0580 -5311> 2014-11-08 18:15:09.118723 7fc5a152e700 5 -- op tracker -- seq: 47077, time: 2014-11-08 18:15:09.118613, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5310> 2014-11-08 18:15:09.118732 7fc5a152e700 5 -- op tracker -- seq: 47077, time: 2014-11-08 18:15:09.118616, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5309> 2014-11-08 18:15:09.118735 7fc5a152e700 5 -- op tracker -- seq: 47077, time: 2014-11-08 18:15:09.118689, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5308> 2014-11-08 18:15:09.118738 7fc5a152e700 5 -- op tracker -- seq: 47077, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5307> 2014-11-08 18:15:09.118796 7fc5ad310700 5 -- op tracker -- seq: 47077, time: 2014-11-08 18:15:09.118796, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5306> 2014-11-08 18:15:09.118808 7fc5ad310700 5 -- op tracker -- seq: 47077, time: 2014-11-08 18:15:09.118808, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5305> 2014-11-08 18:15:09.118855 7fc5ad310700 5 -- op tracker -- seq: 47077, time: 2014-11-08 18:15:09.118855, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 84047b16/1000001a1dd.000001f9/head//4) -5304> 2014-11-08 18:15:09.203701 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 502 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:09.201991) v2 ==== 47+0+0 (3691689526 0 0) 0x1e6ad820 con 0x1eb9d800 -5303> 2014-11-08 18:15:09.203755 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.201991) v2 -- ?+0 0x1d1b41a0 con 0x1eb9d800 -5302> 2014-11-08 18:15:09.204361 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 502 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:09.201991) v2 ==== 47+0+0 (3691689526 0 0) 0x1e6af440 con 0x1eb9c8e0 -5301> 2014-11-08 18:15:09.204375 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.201991) v2 -- ?+0 0x227af260 con 0x1eb9c8e0 -5300> 2014-11-08 18:15:09.297990 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 321 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:09.295607) v2 ==== 47+0+0 (746701687 0 0) 0x239e92c0 con 0x1eb9c780 -5299> 2014-11-08 18:15:09.298038 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.295607) v2 -- ?+0 0x1e6af440 con 0x1eb9c780 -5298> 2014-11-08 18:15:09.298096 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 321 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:09.295607) v2 ==== 47+0+0 (746701687 0 0) 0x224a05a0 con 0x1eb9d6a0 -5297> 2014-11-08 18:15:09.298107 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.295607) v2 -- ?+0 0x1e6ad820 con 0x1eb9d6a0 -5296> 2014-11-08 18:15:09.620523 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8560 ==== MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (3376761273 0 0) 0x24847400 con 0x1c5c0580 -5295> 2014-11-08 18:15:09.620573 7fc5a152e700 5 -- op tracker -- seq: 47078, time: 2014-11-08 18:15:09.523820, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5294> 2014-11-08 18:15:09.620592 7fc5a152e700 5 -- op tracker -- seq: 47078, time: 2014-11-08 18:15:09.523825, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5293> 2014-11-08 18:15:09.620606 7fc5a152e700 5 -- op tracker -- seq: 47078, time: 2014-11-08 18:15:09.620509, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5292> 2014-11-08 18:15:09.620619 7fc5a152e700 5 -- op tracker -- seq: 47078, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5291> 2014-11-08 18:15:09.620668 7fc5afb15700 5 -- op tracker -- seq: 47078, time: 2014-11-08 18:15:09.620668, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5290> 2014-11-08 18:15:09.620852 7fc5afb15700 5 -- op tracker -- seq: 47078, time: 2014-11-08 18:15:09.620852, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(e3857b16/1000002aa3f.0000011d/head//4, version: 3458'5727, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3857b16/1000002aa3f.0000011d/head//4@3458'5727, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c2b57b16/1000002abea.00000043/head//4, version: 3468'5926, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c2b57b16/1000002abea.00000043/head//4@3468'5926, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(88d57b16/1000002986b.00000005/head//4, version: 3033'4543, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88d57b16/1000002986b.00000005/head//4@3033'4543, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5289> 2014-11-08 18:15:09.682482 7fc5c362f700 5 osd.9 5773 tick -5288> 2014-11-08 18:15:09.704878 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 503 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:09.702741) v2 ==== 47+0+0 (3426744514 0 0) 0x1e6af260 con 0x1eb9c8e0 -5287> 2014-11-08 18:15:09.704940 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.702741) v2 -- ?+0 0x239e92c0 con 0x1eb9c8e0 -5286> 2014-11-08 18:15:09.704991 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 503 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:09.702741) v2 ==== 47+0+0 (3426744514 0 0) 0x1e6ad640 con 0x1eb9d800 -5285> 2014-11-08 18:15:09.705003 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.702741) v2 -- ?+0 0x224a05a0 con 0x1eb9d800 -5284> 2014-11-08 18:15:09.754731 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5283> 2014-11-08 18:15:09.754767 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1dec9fe0 con 0x1cae2940 -5282> 2014-11-08 18:15:09.754810 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1dece360 con 0x1cae2680 -5281> 2014-11-08 18:15:09.754822 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1deceae0 con 0x1c5c02c0 -5280> 2014-11-08 18:15:09.754834 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1dec8f00 con 0x1cae0160 -5279> 2014-11-08 18:15:09.754844 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1deca1c0 con 0x1cae0f20 -5278> 2014-11-08 18:15:09.754853 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1dec9a40 con 0x1cae0c60 -5277> 2014-11-08 18:15:09.754864 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x224514a0 con 0x1cae23c0 -5276> 2014-11-08 18:15:09.754873 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x22450f00 con 0x1cae56a0 -5275> 2014-11-08 18:15:09.754884 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1decd0a0 con 0x1cbf9ce0 -5274> 2014-11-08 18:15:09.754895 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1decbde0 con 0x1d603020 -5273> 2014-11-08 18:15:09.754908 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1dec8b40 con 0x1dbd9b80 -5272> 2014-11-08 18:15:09.754917 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1caaf620 con 0x1d235d80 -5271> 2014-11-08 18:15:09.754927 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1decfda0 con 0x1cae1fa0 -5270> 2014-11-08 18:15:09.754936 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1eae1860 con 0x1cae1ce0 -5269> 2014-11-08 18:15:09.754945 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x22a24560 con 0x1cae1340 -5268> 2014-11-08 18:15:09.754954 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x22a25460 con 0x1cae1080 -5267> 2014-11-08 18:15:09.754963 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1ce16ae0 con 0x1cae1a20 -5266> 2014-11-08 18:15:09.754971 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:09.754764) v2 -- ?+0 0x1ce17440 con 0x1cae2c00 -5265> 2014-11-08 18:15:09.755761 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1eac8780 con 0x1cae1fa0 -5264> 2014-11-08 18:15:09.755787 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x24859e00 con 0x1cae1ce0 -5263> 2014-11-08 18:15:09.755825 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x236830c0 con 0x1cae2940 -5262> 2014-11-08 18:15:09.755835 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1d5e4920 con 0x1cae2680 -5261> 2014-11-08 18:15:09.755843 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1c837260 con 0x1c5c02c0 -5260> 2014-11-08 18:15:09.755853 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x2285e540 con 0x1cae0160 -5259> 2014-11-08 18:15:09.755915 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1e531e00 con 0x1cae0c60 -5258> 2014-11-08 18:15:09.755979 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x2495d280 con 0x1cae23c0 -5257> 2014-11-08 18:15:09.755992 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1e6a9a40 con 0x1cae0f20 -5256> 2014-11-08 18:15:09.756752 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x229901e0 con 0x1cae1340 -5255> 2014-11-08 18:15:09.756771 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1d0403c0 con 0x1cae1080 -5254> 2014-11-08 18:15:09.756780 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x1d1b0d20 con 0x1cae1a20 -5253> 2014-11-08 18:15:09.756790 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x2127d0a0 con 0x1cae2c00 -5252> 2014-11-08 18:15:09.756799 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3733 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x25ed6cc0 con 0x1cae56a0 -5251> 2014-11-08 18:15:09.756807 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 320 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x2515b660 con 0x1dbd9b80 -5250> 2014-11-08 18:15:09.756817 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 320 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:09.754764) v2 ==== 47+0+0 (1106070219 0 0) 0x22d54740 con 0x1d235d80 -5249> 2014-11-08 18:15:09.791843 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d069fe0 con 0x1c5c0580 -5248> 2014-11-08 18:15:09.791892 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x213bcec0 con 0x1c5c0580 -5247> 2014-11-08 18:15:09.791900 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22e5f080 con 0x1c5c0580 -5246> 2014-11-08 18:15:09.791909 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(e3857b16/1000002aa3f.0000011d/head//4),PushReplyOp(c2b57b16/1000002abea.00000043/head//4),PushReplyOp(88d57b16/1000002986b.00000005/head//4)]) v2 -- ?+0 0x1d3a8200 con 0x1c5c0580 -5245> 2014-11-08 18:15:09.796292 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8561 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) v3 ==== 757+0+0 (869053704 0 0) 0x23668480 con 0x1c5c0580 -5244> 2014-11-08 18:15:09.796337 7fc5a152e700 5 -- op tracker -- seq: 47079, time: 2014-11-08 18:15:09.796163, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5243> 2014-11-08 18:15:09.796353 7fc5a152e700 5 -- op tracker -- seq: 47079, time: 2014-11-08 18:15:09.796168, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5242> 2014-11-08 18:15:09.796362 7fc5a152e700 5 -- op tracker -- seq: 47079, time: 2014-11-08 18:15:09.796269, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5241> 2014-11-08 18:15:09.796372 7fc5a152e700 5 -- op tracker -- seq: 47079, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5240> 2014-11-08 18:15:09.796418 7fc5ad310700 5 -- op tracker -- seq: 47079, time: 2014-11-08 18:15:09.796418, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5239> 2014-11-08 18:15:09.796429 7fc5ad310700 5 -- op tracker -- seq: 47079, time: 2014-11-08 18:15:09.796429, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5238> 2014-11-08 18:15:09.796644 7fc5a152e700 5 -- op tracker -- seq: 47079, time: 2014-11-08 18:15:09.796644, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 88d57b16/1000002986b.00000005/head//4) -5237> 2014-11-08 18:15:09.922296 7fc5b2b1b700 10 monclient: tick -5236> 2014-11-08 18:15:09.922328 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:14:39.922326) -5235> 2014-11-08 18:15:09.922341 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:15:09.922341; renew after: 2014-11-08 18:16:29.290049) -- no -5234> 2014-11-08 18:15:09.944031 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8562 ==== MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3972335233 0 0) 0x1d3a8200 con 0x1c5c0580 -5233> 2014-11-08 18:15:09.944100 7fc5a152e700 5 -- op tracker -- seq: 47080, time: 2014-11-08 18:15:09.905427, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5232> 2014-11-08 18:15:09.944118 7fc5a152e700 5 -- op tracker -- seq: 47080, time: 2014-11-08 18:15:09.905432, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5231> 2014-11-08 18:15:09.944131 7fc5a152e700 5 -- op tracker -- seq: 47080, time: 2014-11-08 18:15:09.944002, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5230> 2014-11-08 18:15:09.944144 7fc5a152e700 5 -- op tracker -- seq: 47080, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5229> 2014-11-08 18:15:09.944188 7fc5afb15700 5 -- op tracker -- seq: 47080, time: 2014-11-08 18:15:09.944188, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5228> 2014-11-08 18:15:09.946268 7fc5a152e700 5 -- op tracker -- seq: 47080, time: 2014-11-08 18:15:09.946268, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(77767b16/1000002a6f3.00000003/head//4, version: 3458'5258, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(77767b16/1000002a6f3.00000003/head//4@3458'5258, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5227> 2014-11-08 18:15:10.003978 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22e5d280 con 0x1c5c0580 -5226> 2014-11-08 18:15:10.004119 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(77767b16/1000002a6f3.00000003/head//4)]) v2 -- ?+0 0x24847400 con 0x1c5c0580 -5225> 2014-11-08 18:15:10.009073 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8563 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) v3 ==== 757+0+0 (2936553839 0 0) 0x23668480 con 0x1c5c0580 -5224> 2014-11-08 18:15:10.009103 7fc5a152e700 5 -- op tracker -- seq: 47081, time: 2014-11-08 18:15:10.008998, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5223> 2014-11-08 18:15:10.009117 7fc5a152e700 5 -- op tracker -- seq: 47081, time: 2014-11-08 18:15:10.009002, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5222> 2014-11-08 18:15:10.009127 7fc5a152e700 5 -- op tracker -- seq: 47081, time: 2014-11-08 18:15:10.009057, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5221> 2014-11-08 18:15:10.009136 7fc5a152e700 5 -- op tracker -- seq: 47081, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5220> 2014-11-08 18:15:10.009181 7fc5ad310700 5 -- op tracker -- seq: 47081, time: 2014-11-08 18:15:10.009181, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5219> 2014-11-08 18:15:10.009194 7fc5ad310700 5 -- op tracker -- seq: 47081, time: 2014-11-08 18:15:10.009194, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5218> 2014-11-08 18:15:10.009266 7fc5ad310700 5 -- op tracker -- seq: 47081, time: 2014-11-08 18:15:10.009266, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 77767b16/1000002a6f3.00000003/head//4) -5217> 2014-11-08 18:15:10.255414 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5216> 2014-11-08 18:15:10.255446 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1ce141a0 con 0x1cae2940 -5215> 2014-11-08 18:15:10.255472 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x22a27da0 con 0x1cae2680 -5214> 2014-11-08 18:15:10.255482 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x22a27800 con 0x1c5c02c0 -5213> 2014-11-08 18:15:10.255493 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1db1f800 con 0x1cae0160 -5212> 2014-11-08 18:15:10.255503 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1db1da00 con 0x1cae0f20 -5211> 2014-11-08 18:15:10.255513 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1db1c1a0 con 0x1cae0c60 -5210> 2014-11-08 18:15:10.255523 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1d1b5820 con 0x1cae23c0 -5209> 2014-11-08 18:15:10.255533 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1d1b7620 con 0x1cae56a0 -5208> 2014-11-08 18:15:10.255542 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x2290ba20 con 0x1cbf9ce0 -5207> 2014-11-08 18:15:10.255553 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x233fab20 con 0x1d603020 -5206> 2014-11-08 18:15:10.255563 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x23144740 con 0x1dbd9b80 -5205> 2014-11-08 18:15:10.255573 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x23145a00 con 0x1d235d80 -5204> 2014-11-08 18:15:10.255583 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1d047080 con 0x1cae1fa0 -5203> 2014-11-08 18:15:10.255591 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1db1a940 con 0x1cae1ce0 -5202> 2014-11-08 18:15:10.255600 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1d324920 con 0x1cae1340 -5201> 2014-11-08 18:15:10.255609 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x22a20000 con 0x1cae1080 -5200> 2014-11-08 18:15:10.255618 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x1e6a90e0 con 0x1cae1a20 -5199> 2014-11-08 18:15:10.255627 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:10.255443) v2 -- ?+0 0x213be540 con 0x1cae2c00 -5198> 2014-11-08 18:15:10.256096 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x23687440 con 0x1cae2940 -5197> 2014-11-08 18:15:10.256123 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1d5e2b20 con 0x1cae2680 -5196> 2014-11-08 18:15:10.256197 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1c8350a0 con 0x1c5c02c0 -5195> 2014-11-08 18:15:10.256501 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1eac9fe0 con 0x1cae0f20 -5194> 2014-11-08 18:15:10.256516 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1e537080 con 0x1cae0c60 -5193> 2014-11-08 18:15:10.256525 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x229d5a00 con 0x1cae1ce0 -5192> 2014-11-08 18:15:10.256599 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x24959e00 con 0x1cae23c0 -5191> 2014-11-08 18:15:10.256609 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x25ed6360 con 0x1cae56a0 -5190> 2014-11-08 18:15:10.256617 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 321 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x251592c0 con 0x1dbd9b80 -5189> 2014-11-08 18:15:10.256626 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 321 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x22d54ce0 con 0x1d235d80 -5188> 2014-11-08 18:15:10.256634 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x2285da00 con 0x1cae0160 -5187> 2014-11-08 18:15:10.256643 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1eace540 con 0x1cae1fa0 -5186> 2014-11-08 18:15:10.256700 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x229910e0 con 0x1cae1340 -5185> 2014-11-08 18:15:10.256712 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1d044740 con 0x1cae1080 -5184> 2014-11-08 18:15:10.256722 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x1d1b6540 con 0x1cae1a20 -5183> 2014-11-08 18:15:10.256730 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3734 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.255443) v2 ==== 47+0+0 (785321752 0 0) 0x2127e360 con 0x1cae2c00 -5182> 2014-11-08 18:15:10.320511 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3743 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:10.319637) v2 ==== 47+0+0 (1813159970 0 0) 0x225783c0 con 0x1cce3180 -5181> 2014-11-08 18:15:10.320539 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.319637) v2 -- ?+0 0x1e6ad640 con 0x1cce3180 -5180> 2014-11-08 18:15:10.320559 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3743 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:10.319637) v2 ==== 47+0+0 (1813159970 0 0) 0x1e8901e0 con 0x1cce3700 -5179> 2014-11-08 18:15:10.320568 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:10.319637) v2 -- ?+0 0x1e6af260 con 0x1cce3700 -5178> 2014-11-08 18:15:10.593083 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8564 ==== MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (555415702 0 0) 0x24847400 con 0x1c5c0580 -5177> 2014-11-08 18:15:10.593137 7fc5a152e700 5 -- op tracker -- seq: 47082, time: 2014-11-08 18:15:10.475611, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5176> 2014-11-08 18:15:10.593161 7fc5a152e700 5 -- op tracker -- seq: 47082, time: 2014-11-08 18:15:10.475617, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5175> 2014-11-08 18:15:10.593181 7fc5a152e700 5 -- op tracker -- seq: 47082, time: 2014-11-08 18:15:10.593049, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5174> 2014-11-08 18:15:10.593200 7fc5a152e700 5 -- op tracker -- seq: 47082, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5173> 2014-11-08 18:15:10.593251 7fc5afb15700 5 -- op tracker -- seq: 47082, time: 2014-11-08 18:15:10.593251, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5172> 2014-11-08 18:15:10.608179 7fc5a152e700 5 -- op tracker -- seq: 47082, time: 2014-11-08 18:15:10.608178, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(b6677b16/1000002b531.000000c7/head//4, version: 3473'6573, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6677b16/1000002b531.000000c7/head//4@3473'6573, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(39877b16/1000002b2d1.0000008d/head//4, version: 3468'6244, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(39877b16/1000002b2d1.0000008d/head//4@3468'6244, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4ea77b16/10000019b76.0000001b/head//4, version: 397'1180, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4ea77b16/10000019b76.0000001b/head//4@397'1180, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5171> 2014-11-08 18:15:10.682595 7fc5c362f700 5 osd.9 5773 tick -5170> 2014-11-08 18:15:10.755729 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22e58b40 con 0x1c5c0580 -5169> 2014-11-08 18:15:10.755793 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x234e6720 con 0x1c5c0580 -5168> 2014-11-08 18:15:10.755800 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1ce15280 con 0x1c5c0580 -5167> 2014-11-08 18:15:10.755809 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(b6677b16/1000002b531.000000c7/head//4),PushReplyOp(39877b16/1000002b2d1.0000008d/head//4),PushReplyOp(4ea77b16/10000019b76.0000001b/head//4)]) v2 -- ?+0 0x1d3aa800 con 0x1c5c0580 -5166> 2014-11-08 18:15:10.764071 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8565 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) v3 ==== 757+0+0 (2649267002 0 0) 0x2366c800 con 0x1c5c0580 -5165> 2014-11-08 18:15:10.764106 7fc5a152e700 5 -- op tracker -- seq: 47083, time: 2014-11-08 18:15:10.763978, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5164> 2014-11-08 18:15:10.764117 7fc5a152e700 5 -- op tracker -- seq: 47083, time: 2014-11-08 18:15:10.763981, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5163> 2014-11-08 18:15:10.764121 7fc5a152e700 5 -- op tracker -- seq: 47083, time: 2014-11-08 18:15:10.764062, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5162> 2014-11-08 18:15:10.764126 7fc5a152e700 5 -- op tracker -- seq: 47083, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5161> 2014-11-08 18:15:10.764167 7fc5ad310700 5 -- op tracker -- seq: 47083, time: 2014-11-08 18:15:10.764167, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5160> 2014-11-08 18:15:10.764177 7fc5ad310700 5 -- op tracker -- seq: 47083, time: 2014-11-08 18:15:10.764177, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5159> 2014-11-08 18:15:10.764230 7fc5ad310700 5 -- op tracker -- seq: 47083, time: 2014-11-08 18:15:10.764230, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4ea77b16/10000019b76.0000001b/head//4) -5158> 2014-11-08 18:15:11.356058 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -5157> 2014-11-08 18:15:11.356090 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x1d077440 con 0x1cae2940 -5156> 2014-11-08 18:15:11.356116 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x2234cce0 con 0x1cae2680 -5155> 2014-11-08 18:15:11.356126 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x1db1f080 con 0x1c5c02c0 -5154> 2014-11-08 18:15:11.356136 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x1ec24380 con 0x1cae0160 -5153> 2014-11-08 18:15:11.356145 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x23d77620 con 0x1cae0f20 -5152> 2014-11-08 18:15:11.356201 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x45301e0 con 0x1cae0c60 -5151> 2014-11-08 18:15:11.356211 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x233ff800 con 0x1cae23c0 -5150> 2014-11-08 18:15:11.356221 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x2285b660 con 0x1cae56a0 -5149> 2014-11-08 18:15:11.356231 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x2290d0a0 con 0x1cbf9ce0 -5148> 2014-11-08 18:15:11.356241 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x225832a0 con 0x1d603020 -5147> 2014-11-08 18:15:11.356250 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x1ec21c20 con 0x1dbd9b80 -5146> 2014-11-08 18:15:11.356260 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x25ed4560 con 0x1d235d80 -5145> 2014-11-08 18:15:11.356269 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x231450a0 con 0x1cae1fa0 -5144> 2014-11-08 18:15:11.356277 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x23142b20 con 0x1cae1ce0 -5143> 2014-11-08 18:15:11.356286 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x231401e0 con 0x1cae1340 -5142> 2014-11-08 18:15:11.356295 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x1d5e50a0 con 0x1cae1080 -5141> 2014-11-08 18:15:11.356304 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x25ed3660 con 0x1cae1a20 -5140> 2014-11-08 18:15:11.356330 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:11.356086) v2 -- ?+0 0x213b8b40 con 0x1cae2c00 -5139> 2014-11-08 18:15:11.359892 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x1d5e0d20 con 0x1cae2680 -5138> 2014-11-08 18:15:11.359924 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x22e9e540 con 0x1cae23c0 -5137> 2014-11-08 18:15:11.359934 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x25ed50a0 con 0x1cae56a0 -5136> 2014-11-08 18:15:11.359942 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x1eacdfa0 con 0x1cae0f20 -5135> 2014-11-08 18:15:11.359951 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x248585a0 con 0x1cae0c60 -5134> 2014-11-08 18:15:11.359960 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x22995dc0 con 0x1cae1340 -5133> 2014-11-08 18:15:11.359969 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x1d046cc0 con 0x1cae1080 -5132> 2014-11-08 18:15:11.359977 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x2336d820 con 0x1cae1ce0 -5131> 2014-11-08 18:15:11.359986 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x23686ae0 con 0x1cae2940 -5130> 2014-11-08 18:15:11.360158 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x1d1b7da0 con 0x1cae1a20 -5129> 2014-11-08 18:15:11.360171 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x2127cb00 con 0x1cae2c00 -5128> 2014-11-08 18:15:11.360195 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 322 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x2515a580 con 0x1dbd9b80 -5127> 2014-11-08 18:15:11.360204 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 322 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x22d50960 con 0x1d235d80 -5126> 2014-11-08 18:15:11.360212 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x1eacda00 con 0x1cae1fa0 -5125> 2014-11-08 18:15:11.361250 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x2285cec0 con 0x1cae0160 -5124> 2014-11-08 18:15:11.361266 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3735 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:11.356086) v2 ==== 47+0+0 (164403595 0 0) 0x1c832940 con 0x1c5c02c0 -5123> 2014-11-08 18:15:11.578114 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6809/5031236 1605 ==== MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3992800784 0 0) 0x1ce672c0 con 0x21468c60 -5122> 2014-11-08 18:15:11.578154 7fc5a39d2700 5 -- op tracker -- seq: 47084, time: 2014-11-08 18:15:11.578006, event: header_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) -5121> 2014-11-08 18:15:11.578163 7fc5a39d2700 5 -- op tracker -- seq: 47084, time: 2014-11-08 18:15:11.578012, event: throttled, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) -5120> 2014-11-08 18:15:11.578167 7fc5a39d2700 5 -- op tracker -- seq: 47084, time: 2014-11-08 18:15:11.578104, event: all_read, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) -5119> 2014-11-08 18:15:11.578180 7fc5a39d2700 5 -- op tracker -- seq: 47084, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) -5118> 2014-11-08 18:15:11.578228 7fc5acb0f700 5 -- op tracker -- seq: 47084, time: 2014-11-08 18:15:11.578228, event: reached_pg, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) -5117> 2014-11-08 18:15:11.682724 7fc5c362f700 5 osd.9 5773 tick -5116> 2014-11-08 18:15:11.986337 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8566 ==== MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (415270721 0 0) 0x1d3aa800 con 0x1c5c0580 -5115> 2014-11-08 18:15:11.986401 7fc5a152e700 5 -- op tracker -- seq: 47085, time: 2014-11-08 18:15:11.677290, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5114> 2014-11-08 18:15:11.986442 7fc5a152e700 5 -- op tracker -- seq: 47085, time: 2014-11-08 18:15:11.677295, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5113> 2014-11-08 18:15:11.986457 7fc5a152e700 5 -- op tracker -- seq: 47085, time: 2014-11-08 18:15:11.986326, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5112> 2014-11-08 18:15:11.986483 7fc5a152e700 5 -- op tracker -- seq: 47085, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5111> 2014-11-08 18:15:11.986569 7fc5afb15700 5 -- op tracker -- seq: 47085, time: 2014-11-08 18:15:11.986569, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5110> 2014-11-08 18:15:11.986715 7fc5afb15700 5 -- op tracker -- seq: 47085, time: 2014-11-08 18:15:11.986715, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(67c77b16/100000283ea.00000032/head//4, version: 3033'3687, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(67c77b16/100000283ea.00000032/head//4@3033'3687, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b6d87b16/1000002b019.00000098/head//4, version: 4157'7894, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b6d87b16/1000002b019.00000098/head//4@4157'7894, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4f4a7b16/1000002a7db.00000049/head//4, version: 3458'5453, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4f4a7b16/1000002a7db.00000049/head//4@3458'5453, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5109> 2014-11-08 18:15:12.005430 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 504 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:12.003458) v2 ==== 47+0+0 (2404567045 0 0) 0x1e6ad460 con 0x1eb9d800 -5108> 2014-11-08 18:15:12.005485 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:12.003458) v2 -- ?+0 0x225783c0 con 0x1eb9d800 -5107> 2014-11-08 18:15:12.005537 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 504 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:12.003458) v2 ==== 47+0+0 (2404567045 0 0) 0x1e6af080 con 0x1eb9c8e0 -5106> 2014-11-08 18:15:12.005548 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:12.003458) v2 -- ?+0 0x1e8901e0 con 0x1eb9c8e0 -5105> 2014-11-08 18:15:12.137892 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2285fbc0 con 0x1c5c0580 -5104> 2014-11-08 18:15:12.137934 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1d4f1680 con 0x1c5c0580 -5103> 2014-11-08 18:15:12.137941 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1ddcba20 con 0x1c5c0580 -5102> 2014-11-08 18:15:12.137949 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(67c77b16/100000283ea.00000032/head//4),PushReplyOp(b6d87b16/1000002b019.00000098/head//4),PushReplyOp(4f4a7b16/1000002a7db.00000049/head//4)]) v2 -- ?+0 0x1d3a8c00 con 0x1c5c0580 -5101> 2014-11-08 18:15:12.143103 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8567 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) v3 ==== 757+0+0 (1563989667 0 0) 0x2366cc80 con 0x1c5c0580 -5100> 2014-11-08 18:15:12.143135 7fc5a152e700 5 -- op tracker -- seq: 47086, time: 2014-11-08 18:15:12.143014, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5099> 2014-11-08 18:15:12.143143 7fc5a152e700 5 -- op tracker -- seq: 47086, time: 2014-11-08 18:15:12.143016, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5098> 2014-11-08 18:15:12.143146 7fc5a152e700 5 -- op tracker -- seq: 47086, time: 2014-11-08 18:15:12.143095, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5097> 2014-11-08 18:15:12.143149 7fc5a152e700 5 -- op tracker -- seq: 47086, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5096> 2014-11-08 18:15:12.143195 7fc5ad310700 5 -- op tracker -- seq: 47086, time: 2014-11-08 18:15:12.143195, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5095> 2014-11-08 18:15:12.143217 7fc5ad310700 5 -- op tracker -- seq: 47086, time: 2014-11-08 18:15:12.143217, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5094> 2014-11-08 18:15:12.143265 7fc5ad310700 5 -- op tracker -- seq: 47086, time: 2014-11-08 18:15:12.143265, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 4f4a7b16/1000002a7db.00000049/head//4) -5093> 2014-11-08 18:15:12.297000 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8568 ==== MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(995dfcb5/ 1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8864284+0+0 (2701507962 0 0) 0x1d3a8c00 con 0x1c5c0580 -5092> 2014-11-08 18:15:12.297133 7fc5a152e700 5 -- op tracker -- seq: 47087, time: 2014-11-08 18:15:12.176065, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 995dfcb5/1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5091> 2014-11-08 18:15:12.297159 7fc5a152e700 5 -- op tracker -- seq: 47087, time: 2014-11-08 18:15:12.176069, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 995dfcb5/1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5090> 2014-11-08 18:15:12.297180 7fc5a152e700 5 -- op tracker -- seq: 47087, time: 2014-11-08 18:15:12.296982, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 995dfcb5/1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5089> 2014-11-08 18:15:12.297203 7fc5a152e700 5 -- op tracker -- seq: 47087, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(995dfcb5/ 1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5088> 2014-11-08 18:15:12.297260 7fc5b0316700 5 -- op tracker -- seq: 47087, time: 2014-11-08 18:15:12.297260, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 995dfcb5/1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5087> 2014-11-08 18:15:12.297524 7fc5a152e700 5 -- op tracker -- seq: 47087, time: 2014-11-08 18:15:12.297524, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(ee8cfcb5/1000001b30c.00000015/head//4, version: 2829'2859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ee8cfcb5/1000001b30c.00000015/head//4@2829'2859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3f8cfcb5/1000001b50d.00000023/head//4, version: 3033'3284, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3f8cfcb5/1000001b50d.00000023/head//4@3033'3284, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5fcfcb5/1000001a794.00000167/head//4, version: 619'2247, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5fcfcb5/1000001a794.00000167/head//4@619'2247, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(995dfcb5/1000001b381.0000004a/head//4, version: 3033'2906, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(995dfcb5/ 1000001b381.0000004a/head//4@3033'2906, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7bdfcb5/10000014cd0.00000000/head//4, version: 397'607, data_included: [0~471040], data_size: 471040, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(d7bdfcb5/10000014cd0.00000000/head//4@397'607, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:942080, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5086> 2014-11-08 18:15:12.362919 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8569 ==== MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1623784565 0 0) 0x1d3a8c00 con 0x1c5c0580 -5085> 2014-11-08 18:15:12.362963 7fc5a152e700 5 -- op tracker -- seq: 47088, time: 2014-11-08 18:15:12.311553, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5084> 2014-11-08 18:15:12.362976 7fc5a152e700 5 -- op tracker -- seq: 47088, time: 2014-11-08 18:15:12.311559, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5083> 2014-11-08 18:15:12.362982 7fc5a152e700 5 -- op tracker -- seq: 47088, time: 2014-11-08 18:15:12.362901, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5082> 2014-11-08 18:15:12.362988 7fc5a152e700 5 -- op tracker -- seq: 47088, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5081> 2014-11-08 18:15:12.363041 7fc5afb15700 5 -- op tracker -- seq: 47088, time: 2014-11-08 18:15:12.363041, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5080> 2014-11-08 18:15:12.363119 7fc5afb15700 5 -- op tracker -- seq: 47088, time: 2014-11-08 18:15:12.363119, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(141b7b16/10000019b03.000003a3/head//4, version: 397'1033, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(141b7b16/10000019b03.000003a3/head//4@397'1033, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5079> 2014-11-08 18:15:12.459180 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1ddcf9e0 con 0x1c5c0580 -5078> 2014-11-08 18:15:12.459220 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1d077bc0 con 0x1c5c0580 -5077> 2014-11-08 18:15:12.459227 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1d071e00 con 0x1c5c0580 -5076> 2014-11-08 18:15:12.459232 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x23d712c0 con 0x1c5c0580 -5075> 2014-11-08 18:15:12.459238 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x259f2b20 con 0x1c5c0580 -5074> 2014-11-08 18:15:12.459246 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(ee8cfcb5/1000001b30c.00000015/head//4),PushReplyOp(3f8cfcb5/1000001b50d.00000023/head//4),PushReplyOp(5fcfcb5/1000001a794.00000167/head//4),PushReplyOp(995dfcb5/1000001b381.0000004a/head//4),PushReplyOp(d7bdfcb5/10000014cd0.00000000/head//4)]) v2 -- ?+0 0x1d85f600 con 0x1c5c0580 -5073> 2014-11-08 18:15:12.467660 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8570 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) v3 ==== 757+0+0 (349562158 0 0) 0x2366e780 con 0x1c5c0580 -5072> 2014-11-08 18:15:12.467712 7fc5a152e700 5 -- op tracker -- seq: 47089, time: 2014-11-08 18:15:12.467546, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5071> 2014-11-08 18:15:12.467721 7fc5a152e700 5 -- op tracker -- seq: 47089, time: 2014-11-08 18:15:12.467550, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5070> 2014-11-08 18:15:12.467724 7fc5a152e700 5 -- op tracker -- seq: 47089, time: 2014-11-08 18:15:12.467648, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5069> 2014-11-08 18:15:12.467727 7fc5a152e700 5 -- op tracker -- seq: 47089, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5068> 2014-11-08 18:15:12.467781 7fc5adb11700 5 -- op tracker -- seq: 47089, time: 2014-11-08 18:15:12.467781, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5067> 2014-11-08 18:15:12.467795 7fc5adb11700 5 -- op tracker -- seq: 47089, time: 2014-11-08 18:15:12.467794, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5066> 2014-11-08 18:15:12.467845 7fc5adb11700 5 -- op tracker -- seq: 47089, time: 2014-11-08 18:15:12.467844, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d7bdfcb5/10000014cd0.00000000/head//4) -5065> 2014-11-08 18:15:12.664129 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x259f79e0 con 0x1c5c0580 -5064> 2014-11-08 18:15:12.664172 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(141b7b16/10000019b03.000003a3/head//4)]) v2 -- ?+0 0x1d3aa800 con 0x1c5c0580 -5063> 2014-11-08 18:15:12.668120 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8571 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) v3 ==== 757+0+0 (1047640872 0 0) 0x23668900 con 0x1c5c0580 -5062> 2014-11-08 18:15:12.668146 7fc5a152e700 5 -- op tracker -- seq: 47090, time: 2014-11-08 18:15:12.668041, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5061> 2014-11-08 18:15:12.668154 7fc5a152e700 5 -- op tracker -- seq: 47090, time: 2014-11-08 18:15:12.668044, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5060> 2014-11-08 18:15:12.668157 7fc5a152e700 5 -- op tracker -- seq: 47090, time: 2014-11-08 18:15:12.668114, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5059> 2014-11-08 18:15:12.668160 7fc5a152e700 5 -- op tracker -- seq: 47090, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5058> 2014-11-08 18:15:12.668217 7fc5ad310700 5 -- op tracker -- seq: 47090, time: 2014-11-08 18:15:12.668216, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5057> 2014-11-08 18:15:12.668227 7fc5ad310700 5 -- op tracker -- seq: 47090, time: 2014-11-08 18:15:12.668227, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5056> 2014-11-08 18:15:12.668281 7fc5ad310700 5 -- op tracker -- seq: 47090, time: 2014-11-08 18:15:12.668281, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 141b7b16/10000019b03.000003a3/head//4) -5055> 2014-11-08 18:15:12.679196 7fc5acb0f700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6809/5031236 -- MOSDECSubOpReadReply(4.4es0 5773 ECSubReadReply(tid=59254, attrs_read=0)) v1 -- ?+0 0x1c8b8c80 con 0x21468c60 -5054> 2014-11-08 18:15:12.679249 7fc5acb0f700 5 -- op tracker -- seq: 47084, time: 2014-11-08 18:15:12.679249, event: done, op: MOSDECSubOpRead(4.4es1 5773 ECSubRead(tid=59254, to_read={ddb3414e/10000029542.00000037/head//4=0,4194304}, attrs_to_read=)) -5053> 2014-11-08 18:15:12.682894 7fc5c362f700 5 osd.9 5773 tick -5052> 2014-11-08 18:15:12.709030 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3681 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:12.706698) v2 ==== 47+0+0 (682050580 0 0) 0x234e3840 con 0x1cce3440 -5051> 2014-11-08 18:15:12.709079 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:12.706698) v2 -- ?+0 0x1e6af080 con 0x1cce3440 -5050> 2014-11-08 18:15:12.717219 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3681 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:12.706698) v2 ==== 47+0+0 (682050580 0 0) 0x224a7440 con 0x1cce2ec0 -5049> 2014-11-08 18:15:12.717281 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:12.706698) v2 -- ?+0 0x1e6ad460 con 0x1cce2ec0 -5048> 2014-11-08 18:15:12.954194 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3713 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:12.951752) v2 ==== 47+0+0 (3888300622 0 0) 0x1e5312c0 con 0x1c9f8580 -5047> 2014-11-08 18:15:12.954239 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:12.951752) v2 -- ?+0 0x224a7440 con 0x1c9f8580 -5046> 2014-11-08 18:15:12.954428 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3713 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:12.951752) v2 ==== 47+0+0 (3888300622 0 0) 0x227addc0 con 0x1c8b7d20 -5045> 2014-11-08 18:15:12.954441 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:12.951752) v2 -- ?+0 0x234e3840 con 0x1c8b7d20 -5044> 2014-11-08 18:15:13.211740 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3682 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:13.207494) v2 ==== 47+0+0 (819250627 0 0) 0x234e5be0 con 0x1cce3440 -5043> 2014-11-08 18:15:13.211786 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:13.207494) v2 -- ?+0 0x227addc0 con 0x1cce3440 -5042> 2014-11-08 18:15:13.211884 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3682 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:13.207494) v2 ==== 47+0+0 (819250627 0 0) 0x224a5a00 con 0x1cce2ec0 -5041> 2014-11-08 18:15:13.211906 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:13.207494) v2 -- ?+0 0x1e5312c0 con 0x1cce2ec0 -5040> 2014-11-08 18:15:13.223590 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3744 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:13.220569) v2 ==== 47+0+0 (1755659313 0 0) 0x22579a40 con 0x1cce3180 -5039> 2014-11-08 18:15:13.223618 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:13.220569) v2 -- ?+0 0x224a5a00 con 0x1cce3180 -5038> 2014-11-08 18:15:13.223661 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3744 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:13.220569) v2 ==== 47+0+0 (1755659313 0 0) 0x1e897620 con 0x1cce3700 -5037> 2014-11-08 18:15:13.223672 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:13.220569) v2 -- ?+0 0x234e5be0 con 0x1cce3700 -5036> 2014-11-08 18:15:13.298308 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8572 ==== MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294319+0+0 (3778706118 0 0) 0x1d3aa800 con 0x1c5c0580 -5035> 2014-11-08 18:15:13.298364 7fc5a152e700 5 -- op tracker -- seq: 47091, time: 2014-11-08 18:15:13.171672, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5034> 2014-11-08 18:15:13.298388 7fc5a152e700 5 -- op tracker -- seq: 47091, time: 2014-11-08 18:15:13.171677, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5033> 2014-11-08 18:15:13.298409 7fc5a152e700 5 -- op tracker -- seq: 47091, time: 2014-11-08 18:15:13.298274, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5032> 2014-11-08 18:15:13.298427 7fc5a152e700 5 -- op tracker -- seq: 47091, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5031> 2014-11-08 18:15:13.298488 7fc5b0316700 5 -- op tracker -- seq: 47091, time: 2014-11-08 18:15:13.298488, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5030> 2014-11-08 18:15:13.298624 7fc5b0316700 5 -- op tracker -- seq: 47091, time: 2014-11-08 18:15:13.298624, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(22ffcb5/1000001ac6f.00000134/head//4, version: 2829'2688, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(22ffcb5/1000001ac6f.00000134/head//4@2829'2688, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(fe300db5/1000001b4f2.000000dd/head//4, version: 3033'3168, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fe300db5/1000001b4f2.000000dd/head//4@3033'3168, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(faa00db5/10000028fb7.00000000/head//4, version: 3033'4382, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(faa00db5/10000028fb7.00000000/head//4@3033'4382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5029> 2014-11-08 18:15:13.395733 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x259f5a00 con 0x1c5c0580 -5028> 2014-11-08 18:15:13.395768 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x259f01e0 con 0x1c5c0580 -5027> 2014-11-08 18:15:13.395775 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2290bc00 con 0x1c5c0580 -5026> 2014-11-08 18:15:13.464423 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(22ffcb5/1000001ac6f.00000134/head//4),PushReplyOp(fe300db5/1000001b4f2.000000dd/head//4),PushReplyOp(faa00db5/10000028fb7.00000000/head//4)]) v2 -- ?+0 0x2393fe00 con 0x1c5c0580 -5025> 2014-11-08 18:15:13.479796 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8573 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) v3 ==== 757+0+0 (4197650113 0 0) 0x23669b00 con 0x1c5c0580 -5024> 2014-11-08 18:15:13.479839 7fc5a152e700 5 -- op tracker -- seq: 47092, time: 2014-11-08 18:15:13.479697, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5023> 2014-11-08 18:15:13.479847 7fc5a152e700 5 -- op tracker -- seq: 47092, time: 2014-11-08 18:15:13.479701, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5022> 2014-11-08 18:15:13.479850 7fc5a152e700 5 -- op tracker -- seq: 47092, time: 2014-11-08 18:15:13.479789, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5021> 2014-11-08 18:15:13.479854 7fc5a152e700 5 -- op tracker -- seq: 47092, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5020> 2014-11-08 18:15:13.479894 7fc5adb11700 5 -- op tracker -- seq: 47092, time: 2014-11-08 18:15:13.479894, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5019> 2014-11-08 18:15:13.479907 7fc5adb11700 5 -- op tracker -- seq: 47092, time: 2014-11-08 18:15:13.479907, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5018> 2014-11-08 18:15:13.479959 7fc5adb11700 5 -- op tracker -- seq: 47092, time: 2014-11-08 18:15:13.479959, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb faa00db5/10000028fb7.00000000/head//4) -5017> 2014-11-08 18:15:13.606272 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8574 ==== MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8432180+0+0 (3584239548 0 0) 0x2393fe00 con 0x1c5c0580 -5016> 2014-11-08 18:15:13.606327 7fc5a152e700 5 -- op tracker -- seq: 47093, time: 2014-11-08 18:15:13.515283, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5015> 2014-11-08 18:15:13.606353 7fc5a152e700 5 -- op tracker -- seq: 47093, time: 2014-11-08 18:15:13.515288, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5014> 2014-11-08 18:15:13.606372 7fc5a152e700 5 -- op tracker -- seq: 47093, time: 2014-11-08 18:15:13.606261, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5013> 2014-11-08 18:15:13.606392 7fc5a152e700 5 -- op tracker -- seq: 47093, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5012> 2014-11-08 18:15:13.606445 7fc5afb15700 5 -- op tracker -- seq: 47093, time: 2014-11-08 18:15:13.606445, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5011> 2014-11-08 18:15:13.606630 7fc5a152e700 5 -- op tracker -- seq: 47093, time: 2014-11-08 18:15:13.606630, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(966b7b16/1000002a556.00000047/head//4, version: 3033'5053, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(966b7b16/1000002a556.00000047/head//4@3033'5053, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(832c7b16/1000001a5f0.000000c7/head//4, version: 397'1957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(832c7b16/1000001a5f0.000000c7/head//4@397'1957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9efd7b16/10000012b97.00000000/head//4, version: 397'381, data_included: [0~38912], data_size: 38912, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9efd7b16/10000012b97.00000000/head//4@397'381, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:77824, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16fe7b16/1000002a8aa.0000004c/head//4, version: 3458'5531, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16fe7b16/ 1000002a8aa.0000004c/head//4@3458'5531, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(16308b16/10000019a2c.000001c5/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(16308b16/10000019a2c.000001c5/head//4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -5010> 2014-11-08 18:15:13.683048 7fc5c362f700 5 osd.9 5773 tick -5009> 2014-11-08 18:15:13.769863 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25f97260 con 0x1c5c0580 -5008> 2014-11-08 18:15:13.769910 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2515cce0 con 0x1c5c0580 -5007> 2014-11-08 18:15:13.769917 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2515a3a0 con 0x1c5c0580 -5006> 2014-11-08 18:15:13.769922 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x228681e0 con 0x1c5c0580 -5005> 2014-11-08 18:15:13.769928 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2286c1a0 con 0x1c5c0580 -5004> 2014-11-08 18:15:13.769936 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(966b7b16/1000002a556.00000047/head//4),PushReplyOp(832c7b16/1000001a5f0.000000c7/head//4),PushReplyOp(9efd7b16/10000012b97.00000000/head//4),PushReplyOp(16fe7b16/1000002a8aa.0000004c/head//4),PushReplyOp(16308b16/10000019a2c.000001c5/head//4)]) v2 -- ?+0 0x1d3a8c00 con 0x1c5c0580 -5003> 2014-11-08 18:15:13.779252 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8575 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) v3 ==== 757+0+0 (1098067157 0 0) 0x2366de80 con 0x1c5c0580 -5002> 2014-11-08 18:15:13.779272 7fc5a152e700 5 -- op tracker -- seq: 47094, time: 2014-11-08 18:15:13.779196, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -5001> 2014-11-08 18:15:13.779282 7fc5a152e700 5 -- op tracker -- seq: 47094, time: 2014-11-08 18:15:13.779198, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -5000> 2014-11-08 18:15:13.779285 7fc5a152e700 5 -- op tracker -- seq: 47094, time: 2014-11-08 18:15:13.779245, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -4999> 2014-11-08 18:15:13.779290 7fc5a152e700 5 -- op tracker -- seq: 47094, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -4998> 2014-11-08 18:15:13.779323 7fc5ad310700 5 -- op tracker -- seq: 47094, time: 2014-11-08 18:15:13.779323, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -4997> 2014-11-08 18:15:13.779335 7fc5ad310700 5 -- op tracker -- seq: 47094, time: 2014-11-08 18:15:13.779335, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -4996> 2014-11-08 18:15:13.779383 7fc5ad310700 5 -- op tracker -- seq: 47094, time: 2014-11-08 18:15:13.779383, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 16308b16/10000019a2c.000001c5/head//4) -4995> 2014-11-08 18:15:14.042461 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8576 ==== MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (1923973320 0 0) 0x1d3a8c00 con 0x1c5c0580 -4994> 2014-11-08 18:15:14.044633 7fc5a152e700 5 -- op tracker -- seq: 47095, time: 2014-11-08 18:15:13.973958, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4993> 2014-11-08 18:15:14.044736 7fc5a152e700 5 -- op tracker -- seq: 47095, time: 2014-11-08 18:15:13.973963, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4992> 2014-11-08 18:15:14.044756 7fc5a152e700 5 -- op tracker -- seq: 47095, time: 2014-11-08 18:15:14.042446, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4991> 2014-11-08 18:15:14.044775 7fc5a152e700 5 -- op tracker -- seq: 47095, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4990> 2014-11-08 18:15:14.044828 7fc5b0316700 5 -- op tracker -- seq: 47095, time: 2014-11-08 18:15:14.044828, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4989> 2014-11-08 18:15:14.048335 7fc5a152e700 5 -- op tracker -- seq: 47095, time: 2014-11-08 18:15:14.048335, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(7ad00db5/10000019e50.00000061/head//4, version: 397'1622, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7ad00db5/10000019e50.00000061/head//4@397'1622, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2b010db5/1000001ac6e.00000057/head//4, version: 2829'2677, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b010db5/1000001ac6e.00000057/head//4@2829'2677, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97610db5/100000296a7.00000015/head//4, version: 3033'4248, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97610db5/100000296a7.00000015/head//4@3033'4248, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4988> 2014-11-08 18:15:14.598072 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 322 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:14.596337) v2 ==== 47+0+0 (3449453779 0 0) 0x224a1680 con 0x1eb9d6a0 -4987> 2014-11-08 18:15:14.598113 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:14.596337) v2 -- ?+0 0x22579a40 con 0x1eb9d6a0 -4986> 2014-11-08 18:15:14.598746 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 322 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:14.596337) v2 ==== 47+0+0 (3449453779 0 0) 0x239ea760 con 0x1eb9c780 -4985> 2014-11-08 18:15:14.598761 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:14.596337) v2 -- ?+0 0x1e897620 con 0x1eb9c780 -4984> 2014-11-08 18:15:14.603212 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1caabfc0 con 0x1c5c0580 -4983> 2014-11-08 18:15:14.603275 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239edfa0 con 0x1c5c0580 -4982> 2014-11-08 18:15:14.603284 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239ee540 con 0x1c5c0580 -4981> 2014-11-08 18:15:14.603292 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(7ad00db5/10000019e50.00000061/head//4),PushReplyOp(2b010db5/1000001ac6e.00000057/head//4),PushReplyOp(97610db5/100000296a7.00000015/head//4)]) v2 -- ?+0 0x1d3aa800 con 0x1c5c0580 -4980> 2014-11-08 18:15:14.683216 7fc5c362f700 5 osd.9 5773 tick -4979> 2014-11-08 18:15:14.689136 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8577 ==== MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a0a28b16/ 1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (331472362 0 0) 0x1d3a8c00 con 0x1c5c0580 -4978> 2014-11-08 18:15:14.689197 7fc5a152e700 5 -- op tracker -- seq: 47096, time: 2014-11-08 18:15:14.586110, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( a0a28b16/1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4977> 2014-11-08 18:15:14.689227 7fc5a152e700 5 -- op tracker -- seq: 47096, time: 2014-11-08 18:15:14.586115, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a0a28b16/ 1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4976> 2014-11-08 18:15:14.689273 7fc5a152e700 5 -- op tracker -- seq: 47096, time: 2014-11-08 18:15:14.689104, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a0a28b16/ 1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4975> 2014-11-08 18:15:14.689299 7fc5a152e700 5 -- op tracker -- seq: 47096, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a0a28b16/ 1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4974> 2014-11-08 18:15:14.689364 7fc5afb15700 5 -- op tracker -- seq: 47096, time: 2014-11-08 18:15:14.689364, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( a0a28b16/1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4973> 2014-11-08 18:15:14.689532 7fc5afb15700 5 -- op tracker -- seq: 47096, time: 2014-11-08 18:15:14.689532, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(15508b16/10000019b6c.0000008f/head//4, version: 397'1141, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(15508b16/10000019b6c.0000008f/head//4@397'1141, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(51c08b16/10000029837.00000086/head//4, version: 3033'4527, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(51c08b16/10000029837.00000086/head//4@3033'4527, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e2818b16/100000199d0.00000044/head//4, version: 397'837, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e2818b16/100000199d0.00000044/head//4@397'837, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a0a28b16/1000002a253.0000001c/head//4, version: 3033'4753, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a0a28b16/ 1000002a253.0000001c/head//4@3033'4753, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7fa28b16/1000002b526.00000029/head//4, version: 3473'6556, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7fa28b16/1000002b526.00000029/head//4@3473'6556, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4972> 2014-11-08 18:15:14.819883 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ec560 con 0x1c5c0580 -4971> 2014-11-08 18:15:14.819923 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239eeae0 con 0x1c5c0580 -4970> 2014-11-08 18:15:14.819929 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239eb840 con 0x1c5c0580 -4969> 2014-11-08 18:15:14.819936 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ed640 con 0x1c5c0580 -4968> 2014-11-08 18:15:14.819941 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239eb660 con 0x1c5c0580 -4967> 2014-11-08 18:15:14.819948 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(15508b16/10000019b6c.0000008f/head//4),PushReplyOp(51c08b16/10000029837.00000086/head//4),PushReplyOp(e2818b16/100000199d0.00000044/head//4),PushReplyOp(a0a28b16/1000002a253.0000001c/head//4),PushReplyOp(7fa28b16/1000002b526.00000029/head//4)]) v2 -- ?+0 0x1d3ab800 con 0x1c5c0580 -4966> 2014-11-08 18:15:14.826268 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8578 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) v3 ==== 757+0+0 (1006112242 0 0) 0x23668000 con 0x1c5c0580 -4965> 2014-11-08 18:15:14.826283 7fc5a152e700 5 -- op tracker -- seq: 47097, time: 2014-11-08 18:15:14.826225, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4964> 2014-11-08 18:15:14.826291 7fc5a152e700 5 -- op tracker -- seq: 47097, time: 2014-11-08 18:15:14.826227, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4963> 2014-11-08 18:15:14.826294 7fc5a152e700 5 -- op tracker -- seq: 47097, time: 2014-11-08 18:15:14.826263, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4962> 2014-11-08 18:15:14.826297 7fc5a152e700 5 -- op tracker -- seq: 47097, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4961> 2014-11-08 18:15:14.826334 7fc5ad310700 5 -- op tracker -- seq: 47097, time: 2014-11-08 18:15:14.826334, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4960> 2014-11-08 18:15:14.826344 7fc5ad310700 5 -- op tracker -- seq: 47097, time: 2014-11-08 18:15:14.826344, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4959> 2014-11-08 18:15:14.826386 7fc5ad310700 5 -- op tracker -- seq: 47097, time: 2014-11-08 18:15:14.826386, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 7fa28b16/1000002b526.00000029/head//4) -4958> 2014-11-08 18:15:15.017901 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8579 ==== MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (1984883602 0 0) 0x1d3ab800 con 0x1c5c0580 -4957> 2014-11-08 18:15:15.017952 7fc5a152e700 5 -- op tracker -- seq: 47098, time: 2014-11-08 18:15:14.991331, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4956> 2014-11-08 18:15:15.017971 7fc5a152e700 5 -- op tracker -- seq: 47098, time: 2014-11-08 18:15:14.991336, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4955> 2014-11-08 18:15:15.017984 7fc5a152e700 5 -- op tracker -- seq: 47098, time: 2014-11-08 18:15:15.017890, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4954> 2014-11-08 18:15:15.017997 7fc5a152e700 5 -- op tracker -- seq: 47098, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4953> 2014-11-08 18:15:15.018047 7fc5afb15700 5 -- op tracker -- seq: 47098, time: 2014-11-08 18:15:15.018046, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4952> 2014-11-08 18:15:15.018126 7fc5afb15700 5 -- op tracker -- seq: 47098, time: 2014-11-08 18:15:15.018126, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(ba538b16/1000001b31f.0000003c/head//4, version: 2831'2762, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ba538b16/1000001b31f.0000003c/head//4@2831'2762, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4951> 2014-11-08 18:15:15.074337 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ef260 con 0x1c5c0580 -4950> 2014-11-08 18:15:15.192206 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(ba538b16/1000001b31f.0000003c/head//4)]) v2 -- ?+0 0x1d3a8c00 con 0x1c5c0580 -4949> 2014-11-08 18:15:15.197314 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8580 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) v3 ==== 757+0+0 (1412712142 0 0) 0x2366d100 con 0x1c5c0580 -4948> 2014-11-08 18:15:15.197329 7fc5a152e700 5 -- op tracker -- seq: 47099, time: 2014-11-08 18:15:15.197268, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4947> 2014-11-08 18:15:15.197339 7fc5a152e700 5 -- op tracker -- seq: 47099, time: 2014-11-08 18:15:15.197270, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4946> 2014-11-08 18:15:15.197342 7fc5a152e700 5 -- op tracker -- seq: 47099, time: 2014-11-08 18:15:15.197309, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4945> 2014-11-08 18:15:15.197347 7fc5a152e700 5 -- op tracker -- seq: 47099, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4944> 2014-11-08 18:15:15.197379 7fc5ad310700 5 -- op tracker -- seq: 47099, time: 2014-11-08 18:15:15.197379, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4943> 2014-11-08 18:15:15.197391 7fc5ad310700 5 -- op tracker -- seq: 47099, time: 2014-11-08 18:15:15.197391, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4942> 2014-11-08 18:15:15.197435 7fc5ad310700 5 -- op tracker -- seq: 47099, time: 2014-11-08 18:15:15.197435, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb ba538b16/1000001b31f.0000003c/head//4) -4941> 2014-11-08 18:15:15.456724 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -4940> 2014-11-08 18:15:15.456756 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x213bab20 con 0x1cae2940 -4939> 2014-11-08 18:15:15.456782 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x1dbf9680 con 0x1cae2680 -4938> 2014-11-08 18:15:15.456792 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x1dbf8d20 con 0x1c5c02c0 -4937> 2014-11-08 18:15:15.456803 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x1dbfba20 con 0x1cae0160 -4936> 2014-11-08 18:15:15.456813 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x1dbf9860 con 0x1cae0f20 -4935> 2014-11-08 18:15:15.456824 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x23145820 con 0x1cae0c60 -4934> 2014-11-08 18:15:15.456834 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x23142940 con 0x1cae23c0 -4933> 2014-11-08 18:15:15.456844 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x25fc05a0 con 0x1cae56a0 -4932> 2014-11-08 18:15:15.456855 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x23d73480 con 0x1cbf9ce0 -4931> 2014-11-08 18:15:15.456865 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x2257eae0 con 0x1d603020 -4930> 2014-11-08 18:15:15.456875 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x2257b840 con 0x1dbd9b80 -4929> 2014-11-08 18:15:15.456884 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x234e4740 con 0x1d235d80 -4928> 2014-11-08 18:15:15.456894 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x234e1a40 con 0x1cae1fa0 -4927> 2014-11-08 18:15:15.456905 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x2127cb00 con 0x1cae1ce0 -4926> 2014-11-08 18:15:15.456913 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x1d1b7da0 con 0x1cae1340 -4925> 2014-11-08 18:15:15.456922 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x23686ae0 con 0x1cae1080 -4924> 2014-11-08 18:15:15.456937 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x2336d820 con 0x1cae1a20 -4923> 2014-11-08 18:15:15.456947 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:15.456752) v2 -- ?+0 0x1d046cc0 con 0x1cae2c00 -4922> 2014-11-08 18:15:15.457322 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x2127c560 con 0x1cae2c00 -4921> 2014-11-08 18:15:15.457352 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x22e9a3a0 con 0x1cae23c0 -4920> 2014-11-08 18:15:15.457363 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x1eacf080 con 0x1cae1fa0 -4919> 2014-11-08 18:15:15.457944 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x1d0401e0 con 0x1cae1080 -4918> 2014-11-08 18:15:15.457962 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x23369c20 con 0x1cae1ce0 -4917> 2014-11-08 18:15:15.457971 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x1d1b5640 con 0x1cae1a20 -4916> 2014-11-08 18:15:15.457979 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x23683480 con 0x1cae2940 -4915> 2014-11-08 18:15:15.457988 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x1d5e0000 con 0x1cae2680 -4914> 2014-11-08 18:15:15.457996 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x25ed1680 con 0x1cae56a0 -4913> 2014-11-08 18:15:15.458004 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 323 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x2515bfc0 con 0x1dbd9b80 -4912> 2014-11-08 18:15:15.458013 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 323 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x1eae2940 con 0x1d235d80 -4911> 2014-11-08 18:15:15.458022 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x1c834ec0 con 0x1c5c02c0 -4910> 2014-11-08 18:15:15.458030 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x2285e720 con 0x1cae0160 -4909> 2014-11-08 18:15:15.458038 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x22d82940 con 0x1cae0f20 -4908> 2014-11-08 18:15:15.458047 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x2485ddc0 con 0x1cae0c60 -4907> 2014-11-08 18:15:15.458054 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3736 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:15.456752) v2 ==== 47+0+0 (142395601 0 0) 0x25fd1a40 con 0x1cae1340 -4906> 2014-11-08 18:15:15.683327 7fc5c362f700 5 osd.9 5773 tick -4905> 2014-11-08 18:15:16.124792 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3745 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:16.123241) v2 ==== 47+0+0 (2057366478 0 0) 0x2257a760 con 0x1cce3180 -4904> 2014-11-08 18:15:16.124817 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:16.123241) v2 -- ?+0 0x224a1680 con 0x1cce3180 -4903> 2014-11-08 18:15:16.124840 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3745 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:16.123241) v2 ==== 47+0+0 (2057366478 0 0) 0x1e897bc0 con 0x1cce3700 -4902> 2014-11-08 18:15:16.124849 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:16.123241) v2 -- ?+0 0x239ea760 con 0x1cce3700 -4901> 2014-11-08 18:15:16.179748 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8581 ==== MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/ 1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490148+0+0 (3063031501 0 0) 0x1d3a8c00 con 0x1c5c0580 -4900> 2014-11-08 18:15:16.179814 7fc5a152e700 5 -- op tracker -- seq: 47100, time: 2014-11-08 18:15:16.067176, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4899> 2014-11-08 18:15:16.179845 7fc5a152e700 5 -- op tracker -- seq: 47100, time: 2014-11-08 18:15:16.067180, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4898> 2014-11-08 18:15:16.179870 7fc5a152e700 5 -- op tracker -- seq: 47100, time: 2014-11-08 18:15:16.179736, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4897> 2014-11-08 18:15:16.179894 7fc5a152e700 5 -- op tracker -- seq: 47100, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/ 1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4896> 2014-11-08 18:15:16.179949 7fc5afb15700 5 -- op tracker -- seq: 47100, time: 2014-11-08 18:15:16.179949, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4895> 2014-11-08 18:15:16.190174 7fc5a152e700 5 -- op tracker -- seq: 47100, time: 2014-11-08 18:15:16.190173, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(ad938b16/1000001a9a0.000000f9/head//4, version: 2829'2260, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ad938b16/1000001a9a0.000000f9/head//4@2829'2260, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(cf448b16/10000029508.00000003/head//4, version: 3033'4078, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(cf448b16/10000029508.00000003/head//4@3033'4078, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(91748b16/1000001b482.0000001c/head//4, version: 3033'2973, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(91748b16/1000001b482.0000001c/head//4@3033'2973, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2f758b16/1000002bb05.00000684/head//4, version: 3473'7432, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2f758b16/ 1000002bb05.00000684/head//4@3473'7432, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(54a58b16/1000002bcea.00000010/head//4, version: 4864'8212, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(54a58b16/1000002bcea.00000010/head//4@4864'8212, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4894> 2014-11-08 18:15:16.352155 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239eab20 con 0x1c5c0580 -4893> 2014-11-08 18:15:16.352196 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239efda0 con 0x1c5c0580 -4892> 2014-11-08 18:15:16.352203 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ee900 con 0x1c5c0580 -4891> 2014-11-08 18:15:16.352208 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ef440 con 0x1c5c0580 -4890> 2014-11-08 18:15:16.352214 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239e8f00 con 0x1c5c0580 -4889> 2014-11-08 18:15:16.352221 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(ad938b16/1000001a9a0.000000f9/head//4),PushReplyOp(cf448b16/10000029508.00000003/head//4),PushReplyOp(91748b16/1000001b482.0000001c/head//4),PushReplyOp(2f758b16/1000002bb05.00000684/head//4),PushReplyOp(54a58b16/1000002bcea.00000010/head//4)]) v2 -- ?+0 0x1d3ab800 con 0x1c5c0580 -4888> 2014-11-08 18:15:16.360295 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8582 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) v3 ==== 757+0+0 (24498891 0 0) 0x2366b180 con 0x1c5c0580 -4887> 2014-11-08 18:15:16.360316 7fc5a152e700 5 -- op tracker -- seq: 47101, time: 2014-11-08 18:15:16.360222, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4886> 2014-11-08 18:15:16.360324 7fc5a152e700 5 -- op tracker -- seq: 47101, time: 2014-11-08 18:15:16.360226, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4885> 2014-11-08 18:15:16.360327 7fc5a152e700 5 -- op tracker -- seq: 47101, time: 2014-11-08 18:15:16.360289, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4884> 2014-11-08 18:15:16.360330 7fc5a152e700 5 -- op tracker -- seq: 47101, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4883> 2014-11-08 18:15:16.360368 7fc5ad310700 5 -- op tracker -- seq: 47101, time: 2014-11-08 18:15:16.360368, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4882> 2014-11-08 18:15:16.360379 7fc5ad310700 5 -- op tracker -- seq: 47101, time: 2014-11-08 18:15:16.360379, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4881> 2014-11-08 18:15:16.360425 7fc5ad310700 5 -- op tracker -- seq: 47101, time: 2014-11-08 18:15:16.360425, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 54a58b16/1000002bcea.00000010/head//4) -4880> 2014-11-08 18:15:16.624836 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3746 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:16.623884) v2 ==== 47+0+0 (2120802207 0 0) 0x1e8921c0 con 0x1cce3700 -4879> 2014-11-08 18:15:16.624862 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:16.623884) v2 -- ?+0 0x1e897bc0 con 0x1cce3700 -4878> 2014-11-08 18:15:16.624899 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3746 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:16.623884) v2 ==== 47+0+0 (2120802207 0 0) 0x2257ecc0 con 0x1cce3180 -4877> 2014-11-08 18:15:16.624909 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:16.623884) v2 -- ?+0 0x2257a760 con 0x1cce3180 -4876> 2014-11-08 18:15:16.683480 7fc5c362f700 5 osd.9 5773 tick -4875> 2014-11-08 18:15:16.711041 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3683 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:16.708297) v2 ==== 47+0+0 (2946188946 0 0) 0x224a7da0 con 0x1cce2ec0 -4874> 2014-11-08 18:15:16.711076 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:16.708297) v2 -- ?+0 0x2257ecc0 con 0x1cce2ec0 -4873> 2014-11-08 18:15:16.711032 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3683 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:16.708297) v2 ==== 47+0+0 (2946188946 0 0) 0x234e7440 con 0x1cce3440 -4872> 2014-11-08 18:15:16.711127 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:16.708297) v2 -- ?+0 0x1e8921c0 con 0x1cce3440 -4871> 2014-11-08 18:15:17.125523 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3747 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:17.124428) v2 ==== 47+0+0 (2381883119 0 0) 0x2234d820 con 0x1cce3180 -4870> 2014-11-08 18:15:17.125549 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.124428) v2 -- ?+0 0x224a7da0 con 0x1cce3180 -4869> 2014-11-08 18:15:17.125588 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3747 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:17.124428) v2 ==== 47+0+0 (2381883119 0 0) 0x1e890f00 con 0x1cce3700 -4868> 2014-11-08 18:15:17.125599 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.124428) v2 -- ?+0 0x234e7440 con 0x1cce3700 -4867> 2014-11-08 18:15:17.157317 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1711 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -4866> 2014-11-08 18:15:17.157344 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x22995dc0 con 0x1cae2940 -4865> 2014-11-08 18:15:17.157365 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x248585a0 con 0x1cae2680 -4864> 2014-11-08 18:15:17.157374 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x1eacdfa0 con 0x1c5c02c0 -4863> 2014-11-08 18:15:17.157383 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x25ed50a0 con 0x1cae0160 -4862> 2014-11-08 18:15:17.157392 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x22e9e540 con 0x1cae0f20 -4861> 2014-11-08 18:15:17.157401 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x1d5e0d20 con 0x1cae0c60 -4860> 2014-11-08 18:15:17.157409 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x2127e360 con 0x1cae23c0 -4859> 2014-11-08 18:15:17.157419 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x1d1b6540 con 0x1cae56a0 -4858> 2014-11-08 18:15:17.157428 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x1d044740 con 0x1cbf9ce0 -4857> 2014-11-08 18:15:17.157437 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x229910e0 con 0x1d603020 -4856> 2014-11-08 18:15:17.157447 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x1eace540 con 0x1dbd9b80 -4855> 2014-11-08 18:15:17.157456 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x2285da00 con 0x1d235d80 -4854> 2014-11-08 18:15:17.157466 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x22d54ce0 con 0x1cae1fa0 -4853> 2014-11-08 18:15:17.157475 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x251592c0 con 0x1cae1ce0 -4852> 2014-11-08 18:15:17.157483 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x25ed6360 con 0x1cae1340 -4851> 2014-11-08 18:15:17.157492 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x24959e00 con 0x1cae1080 -4850> 2014-11-08 18:15:17.157505 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x229d5a00 con 0x1cae1a20 -4849> 2014-11-08 18:15:17.157515 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:17.157341) v2 -- ?+0 0x1e537080 con 0x1cae2c00 -4848> 2014-11-08 18:15:17.158149 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x22e9f260 con 0x1cae23c0 -4847> 2014-11-08 18:15:17.158173 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x22d85820 con 0x1cae0f20 -4846> 2014-11-08 18:15:17.158182 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x23926cc0 con 0x1cae0c60 -4845> 2014-11-08 18:15:17.158191 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x1eac9a40 con 0x1cae1fa0 -4844> 2014-11-08 18:15:17.158292 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x1d5e12c0 con 0x1cae2680 -4843> 2014-11-08 18:15:17.158319 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x23369e00 con 0x1cae1ce0 -4842> 2014-11-08 18:15:17.158329 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x23682940 con 0x1cae2940 -4841> 2014-11-08 18:15:17.158338 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 324 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x1dbff800 con 0x1dbd9b80 -4840> 2014-11-08 18:15:17.158348 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x1c832760 con 0x1c5c02c0 -4839> 2014-11-08 18:15:17.158357 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x2285f620 con 0x1cae0160 -4838> 2014-11-08 18:15:17.158402 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x25ed79e0 con 0x1cae56a0 -4837> 2014-11-08 18:15:17.158417 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 324 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x1eae4740 con 0x1d235d80 -4836> 2014-11-08 18:15:17.158426 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x212781e0 con 0x1cae2c00 -4835> 2014-11-08 18:15:17.158497 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x2257e720 con 0x1cae1a20 -4834> 2014-11-08 18:15:17.158703 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x25fd0b40 con 0x1cae1340 -4833> 2014-11-08 18:15:17.158718 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3737 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.157341) v2 ==== 47+0+0 (2295978651 0 0) 0x1d044b00 con 0x1cae1080 -4832> 2014-11-08 18:15:17.198271 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8583 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) v3 ==== 757+0+0 (2104204381 0 0) 0x2366bf00 con 0x1c5c0580 -4831> 2014-11-08 18:15:17.198303 7fc5a152e700 5 -- op tracker -- seq: 47102, time: 2014-11-08 18:15:17.198170, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4830> 2014-11-08 18:15:17.198312 7fc5a152e700 5 -- op tracker -- seq: 47102, time: 2014-11-08 18:15:17.198175, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4829> 2014-11-08 18:15:17.198317 7fc5a152e700 5 -- op tracker -- seq: 47102, time: 2014-11-08 18:15:17.198263, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4828> 2014-11-08 18:15:17.198321 7fc5a152e700 5 -- op tracker -- seq: 47102, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4827> 2014-11-08 18:15:17.198357 7fc5adb11700 5 -- op tracker -- seq: 47102, time: 2014-11-08 18:15:17.198357, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4826> 2014-11-08 18:15:17.198370 7fc5adb11700 5 -- op tracker -- seq: 47102, time: 2014-11-08 18:15:17.198370, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4825> 2014-11-08 18:15:17.198422 7fc5adb11700 5 -- op tracker -- seq: 47102, time: 2014-11-08 18:15:17.198422, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 97610db5/100000296a7.00000015/head//4) -4824> 2014-11-08 18:15:17.509910 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8584 ==== MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d98b16/ 10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8438405+0+0 (1450608292 0 0) 0x1d3ab800 con 0x1c5c0580 -4823> 2014-11-08 18:15:17.509977 7fc5a152e700 5 -- op tracker -- seq: 47103, time: 2014-11-08 18:15:17.355090, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( c8d98b16/10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4822> 2014-11-08 18:15:17.510008 7fc5a152e700 5 -- op tracker -- seq: 47103, time: 2014-11-08 18:15:17.355094, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( c8d98b16/10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4821> 2014-11-08 18:15:17.510032 7fc5a152e700 5 -- op tracker -- seq: 47103, time: 2014-11-08 18:15:17.509897, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( c8d98b16/10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4820> 2014-11-08 18:15:17.510058 7fc5a152e700 5 -- op tracker -- seq: 47103, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d98b16/ 10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4819> 2014-11-08 18:15:17.510116 7fc5afb15700 5 -- op tracker -- seq: 47103, time: 2014-11-08 18:15:17.510115, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( c8d98b16/10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4818> 2014-11-08 18:15:17.518896 7fc5a152e700 5 -- op tracker -- seq: 47103, time: 2014-11-08 18:15:17.518896, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(38b68b16/1000001b288.00000062/head//4, version: 2829'2692, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(38b68b16/1000001b288.00000062/head//4@2829'2692, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f8e78b16/1000002b5fa.000000a1/head//4, version: 3473'6702, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f8e78b16/1000002b5fa.000000a1/head//4@3473'6702, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(97298b16/1000002b03c.00000060/head//4, version: 4157'7914, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(97298b16/1000002b03c.00000060/head//4@4157'7914, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d98b16/10000019726.00000019/head//4, version: 397'733, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d98b16/ 10000019726.00000019/head//4@397'733, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(116a8b16/1000000f01c.00000000/head//4, version: 397'382, data_included: [0~45056], data_size: 45056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(116a8b16/1000000f01c.00000000/head//4@397'382, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:90112, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4817> 2014-11-08 18:15:17.667308 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239e9e00 con 0x1c5c0580 -4816> 2014-11-08 18:15:17.667342 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239edbe0 con 0x1c5c0580 -4815> 2014-11-08 18:15:17.667349 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ef080 con 0x1c5c0580 -4814> 2014-11-08 18:15:17.667354 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ed0a0 con 0x1c5c0580 -4813> 2014-11-08 18:15:17.667359 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239e94a0 con 0x1c5c0580 -4812> 2014-11-08 18:15:17.667367 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(38b68b16/1000001b288.00000062/head//4),PushReplyOp(f8e78b16/1000002b5fa.000000a1/head//4),PushReplyOp(97298b16/1000002b03c.00000060/head//4),PushReplyOp(c8d98b16/10000019726.00000019/head//4),PushReplyOp(116a8b16/1000000f01c.00000000/head//4)]) v2 -- ?+0 0x1d3ad000 con 0x1c5c0580 -4811> 2014-11-08 18:15:17.671918 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8585 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) v3 ==== 757+0+0 (3763202531 0 0) 0x2366ec00 con 0x1c5c0580 -4810> 2014-11-08 18:15:17.671934 7fc5a152e700 5 -- op tracker -- seq: 47104, time: 2014-11-08 18:15:17.671872, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4809> 2014-11-08 18:15:17.671942 7fc5a152e700 5 -- op tracker -- seq: 47104, time: 2014-11-08 18:15:17.671875, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4808> 2014-11-08 18:15:17.671946 7fc5a152e700 5 -- op tracker -- seq: 47104, time: 2014-11-08 18:15:17.671914, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4807> 2014-11-08 18:15:17.671949 7fc5a152e700 5 -- op tracker -- seq: 47104, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4806> 2014-11-08 18:15:17.671982 7fc5ad310700 5 -- op tracker -- seq: 47104, time: 2014-11-08 18:15:17.671981, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4805> 2014-11-08 18:15:17.671994 7fc5ad310700 5 -- op tracker -- seq: 47104, time: 2014-11-08 18:15:17.671994, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4804> 2014-11-08 18:15:17.672039 7fc5ad310700 5 -- op tracker -- seq: 47104, time: 2014-11-08 18:15:17.672038, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 116a8b16/1000000f01c.00000000/head//4) -4803> 2014-11-08 18:15:17.683580 7fc5c362f700 5 osd.9 5773 tick -4802> 2014-11-08 18:15:17.906735 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 505 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:17.904328) v2 ==== 47+0+0 (3558304236 0 0) 0x1e6ad280 con 0x1eb9d800 -4801> 2014-11-08 18:15:17.906759 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.904328) v2 -- ?+0 0x2234d820 con 0x1eb9d800 -4800> 2014-11-08 18:15:17.906830 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 505 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:17.904328) v2 ==== 47+0+0 (3558304236 0 0) 0x1e6aeea0 con 0x1eb9c8e0 -4799> 2014-11-08 18:15:17.906842 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:17.904328) v2 -- ?+0 0x1e890f00 con 0x1eb9c8e0 -4798> 2014-11-08 18:15:18.115030 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8586 ==== MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2566781103 0 0) 0x1d3ad000 con 0x1c5c0580 -4797> 2014-11-08 18:15:18.115076 7fc5a152e700 5 -- op tracker -- seq: 47105, time: 2014-11-08 18:15:18.094834, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4796> 2014-11-08 18:15:18.115095 7fc5a152e700 5 -- op tracker -- seq: 47105, time: 2014-11-08 18:15:18.094839, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4795> 2014-11-08 18:15:18.115107 7fc5a152e700 5 -- op tracker -- seq: 47105, time: 2014-11-08 18:15:18.115005, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4794> 2014-11-08 18:15:18.115119 7fc5a152e700 5 -- op tracker -- seq: 47105, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4793> 2014-11-08 18:15:18.115165 7fc5afb15700 5 -- op tracker -- seq: 47105, time: 2014-11-08 18:15:18.115165, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4792> 2014-11-08 18:15:18.115262 7fc5a152e700 5 -- op tracker -- seq: 47105, time: 2014-11-08 18:15:18.115262, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(817a8b16/1000002b105.000000a3/head//4, version: 4157'8014, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(817a8b16/1000002b105.000000a3/head//4@4157'8014, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4791> 2014-11-08 18:15:18.162087 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x239ea1c0 con 0x1c5c0580 -4790> 2014-11-08 18:15:18.219819 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8587 ==== MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (2090730649 0 0) 0x1d3ad000 con 0x1c5c0580 -4789> 2014-11-08 18:15:18.219888 7fc5a152e700 5 -- op tracker -- seq: 47106, time: 2014-11-08 18:15:18.117329, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4788> 2014-11-08 18:15:18.219915 7fc5a152e700 5 -- op tracker -- seq: 47106, time: 2014-11-08 18:15:18.117331, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4787> 2014-11-08 18:15:18.219975 7fc5a152e700 5 -- op tracker -- seq: 47106, time: 2014-11-08 18:15:18.219789, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4786> 2014-11-08 18:15:18.219994 7fc5a152e700 5 -- op tracker -- seq: 47106, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4785> 2014-11-08 18:15:18.220049 7fc5adb11700 5 -- op tracker -- seq: 47106, time: 2014-11-08 18:15:18.220049, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4784> 2014-11-08 18:15:18.220203 7fc5adb11700 5 -- op tracker -- seq: 47106, time: 2014-11-08 18:15:18.220202, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(75e10db5/1000002b61a.00000077/head//4, version: 3473'6652, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(75e10db5/1000002b61a.00000077/head//4@3473'6652, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8a120db5/1000001a66d.000000ea/head//4, version: 397'2155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8a120db5/1000001a66d.000000ea/head//4@397'2155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f5a20db5/1000002aa61.00000033/head//4, version: 3458'5697, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f5a20db5/1000002aa61.00000033/head//4@3458'5697, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4783> 2014-11-08 18:15:18.254787 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3714 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:18.252380) v2 ==== 47+0+0 (3853520751 0 0) 0x1e533a20 con 0x1c9f8580 -4782> 2014-11-08 18:15:18.254815 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:18.252380) v2 -- ?+0 0x1e6ad280 con 0x1c9f8580 -4781> 2014-11-08 18:15:18.254839 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3714 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:18.252380) v2 ==== 47+0+0 (3853520751 0 0) 0x1eae2ee0 con 0x1c8b7d20 -4780> 2014-11-08 18:15:18.254849 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:18.252380) v2 -- ?+0 0x1e6aeea0 con 0x1c8b7d20 -4779> 2014-11-08 18:15:18.288305 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239eb480 con 0x1c5c0580 -4778> 2014-11-08 18:15:18.288340 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239e83c0 con 0x1c5c0580 -4777> 2014-11-08 18:15:18.288443 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239ed820 con 0x1c5c0580 -4776> 2014-11-08 18:15:18.683691 7fc5c362f700 5 osd.9 5773 tick -4775> 2014-11-08 18:15:18.700220 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 323 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:18.697478) v2 ==== 47+0+0 (306853185 0 0) 0x239eda00 con 0x1eb9c780 -4774> 2014-11-08 18:15:18.700253 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:18.697478) v2 -- ?+0 0x1eae2ee0 con 0x1eb9c780 -4773> 2014-11-08 18:15:18.700279 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 323 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:18.697478) v2 ==== 47+0+0 (306853185 0 0) 0x246ce180 con 0x1eb9d6a0 -4772> 2014-11-08 18:15:18.700288 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:18.697478) v2 -- ?+0 0x1e533a20 con 0x1eb9d6a0 -4771> 2014-11-08 18:15:18.826328 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3748 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:18.825341) v2 ==== 47+0+0 (3429169909 0 0) 0x22348f00 con 0x1cce3180 -4770> 2014-11-08 18:15:18.826369 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:18.825341) v2 -- ?+0 0x246ce180 con 0x1cce3180 -4769> 2014-11-08 18:15:18.826412 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3748 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:18.825341) v2 ==== 47+0+0 (3429169909 0 0) 0x2127f9e0 con 0x1cce3700 -4768> 2014-11-08 18:15:18.826424 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:18.825341) v2 -- ?+0 0x239eda00 con 0x1cce3700 -4767> 2014-11-08 18:15:19.355842 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3715 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:19.353161) v2 ==== 47+0+0 (443680030 0 0) 0x1dbf8b40 con 0x1c9f8580 -4766> 2014-11-08 18:15:19.355892 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:19.353161) v2 -- ?+0 0x22348f00 con 0x1c9f8580 -4765> 2014-11-08 18:15:19.355842 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3715 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:19.353161) v2 ==== 47+0+0 (443680030 0 0) 0x1eae32a0 con 0x1c8b7d20 -4764> 2014-11-08 18:15:19.355943 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:19.353161) v2 -- ?+0 0x2127f9e0 con 0x1c8b7d20 -4763> 2014-11-08 18:15:19.611649 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3684 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:19.608969) v2 ==== 47+0+0 (1661718004 0 0) 0x224a2940 con 0x1cce2ec0 -4762> 2014-11-08 18:15:19.611699 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:19.608969) v2 -- ?+0 0x1dbf8b40 con 0x1cce2ec0 -4761> 2014-11-08 18:15:19.611641 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3684 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:19.608969) v2 ==== 47+0+0 (1661718004 0 0) 0x234e5fa0 con 0x1cce3440 -4760> 2014-11-08 18:15:19.611752 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:19.608969) v2 -- ?+0 0x1eae32a0 con 0x1cce3440 -4759> 2014-11-08 18:15:19.683795 7fc5c362f700 5 osd.9 5773 tick -4758> 2014-11-08 18:15:19.922427 7fc5b2b1b700 10 monclient: tick -4757> 2014-11-08 18:15:19.922454 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:14:49.922452) -4756> 2014-11-08 18:15:19.922467 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:15:19.922466; renew after: 2014-11-08 18:16:29.290049) -- no -4755> 2014-11-08 18:15:20.057944 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -4754> 2014-11-08 18:15:20.057977 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1eac9fe0 con 0x1cae2940 -4753> 2014-11-08 18:15:20.058005 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1c8350a0 con 0x1cae2680 -4752> 2014-11-08 18:15:20.058016 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1d5e2b20 con 0x1c5c02c0 -4751> 2014-11-08 18:15:20.058027 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x23687440 con 0x1cae0160 -4750> 2014-11-08 18:15:20.058038 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x22d54740 con 0x1cae0f20 -4749> 2014-11-08 18:15:20.058048 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x2515b660 con 0x1cae0c60 -4748> 2014-11-08 18:15:20.058058 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x25ed6cc0 con 0x1cae23c0 -4747> 2014-11-08 18:15:20.058069 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x2127d0a0 con 0x1cae56a0 -4746> 2014-11-08 18:15:20.058079 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1d1b0d20 con 0x1cbf9ce0 -4745> 2014-11-08 18:15:20.058088 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1d0403c0 con 0x1d603020 -4744> 2014-11-08 18:15:20.058097 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x229901e0 con 0x1dbd9b80 -4743> 2014-11-08 18:15:20.058107 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1e6a9a40 con 0x1d235d80 -4742> 2014-11-08 18:15:20.058116 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x2495d280 con 0x1cae1fa0 -4741> 2014-11-08 18:15:20.058125 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1e531e00 con 0x1cae1ce0 -4740> 2014-11-08 18:15:20.058134 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x2285e540 con 0x1cae1340 -4739> 2014-11-08 18:15:20.058144 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1c837260 con 0x1cae1080 -4738> 2014-11-08 18:15:20.058153 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x1d5e4920 con 0x1cae1a20 -4737> 2014-11-08 18:15:20.058163 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:20.057974) v2 -- ?+0 0x236830c0 con 0x1cae2c00 -4736> 2014-11-08 18:15:20.058472 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x1c834b00 con 0x1c5c02c0 -4735> 2014-11-08 18:15:20.058536 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x2127eea0 con 0x1cae2c00 -4734> 2014-11-08 18:15:20.058577 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x22e9a580 con 0x1cae23c0 -4733> 2014-11-08 18:15:20.058681 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x22d87620 con 0x1cae0f20 -4732> 2014-11-08 18:15:20.058771 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x23927260 con 0x1cae0c60 -4731> 2014-11-08 18:15:20.058784 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x1eac9e00 con 0x1cae1fa0 -4730> 2014-11-08 18:15:20.058970 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x1d5e5a00 con 0x1cae2680 -4729> 2014-11-08 18:15:20.058983 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 325 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x1ce103c0 con 0x1dbd9b80 -4728> 2014-11-08 18:15:20.058993 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 325 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x1eae7080 con 0x1d235d80 -4727> 2014-11-08 18:15:20.059002 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x22579fe0 con 0x1cae1a20 -4726> 2014-11-08 18:15:20.059010 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x23687620 con 0x1cae2940 -4725> 2014-11-08 18:15:20.059018 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x227ce540 con 0x1cae1ce0 -4724> 2014-11-08 18:15:20.059034 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x1d045dc0 con 0x1cae1080 -4723> 2014-11-08 18:15:20.059050 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x25fd7620 con 0x1cae1340 -4722> 2014-11-08 18:15:20.061116 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x2285b840 con 0x1cae0160 -4721> 2014-11-08 18:15:20.061137 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3738 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.057974) v2 ==== 47+0+0 (2025765475 0 0) 0x22997800 con 0x1cae56a0 -4720> 2014-11-08 18:15:20.207470 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 506 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:20.205078) v2 ==== 47+0+0 (2742522945 0 0) 0x1e6aecc0 con 0x1eb9c8e0 -4719> 2014-11-08 18:15:20.207512 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.205078) v2 -- ?+0 0x234e5fa0 con 0x1eb9c8e0 -4718> 2014-11-08 18:15:20.207714 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 506 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:20.205078) v2 ==== 47+0+0 (2742522945 0 0) 0x1e6ad0a0 con 0x1eb9d800 -4717> 2014-11-08 18:15:20.207727 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:20.205078) v2 -- ?+0 0x224a2940 con 0x1eb9d800 -4716> 2014-11-08 18:15:20.524665 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(75e10db5/1000002b61a.00000077/head//4),PushReplyOp(8a120db5/1000001a66d.000000ea/head//4),PushReplyOp(f5a20db5/1000002aa61.00000033/head//4)]) v2 -- ?+0 0x2393f400 con 0x1c5c0580 -4715> 2014-11-08 18:15:20.528065 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8588 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) v3 ==== 757+0+0 (873378419 0 0) 0x1e805580 con 0x1c5c0580 -4714> 2014-11-08 18:15:20.528093 7fc5a152e700 5 -- op tracker -- seq: 47107, time: 2014-11-08 18:15:20.527985, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4713> 2014-11-08 18:15:20.528102 7fc5a152e700 5 -- op tracker -- seq: 47107, time: 2014-11-08 18:15:20.527989, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4712> 2014-11-08 18:15:20.528105 7fc5a152e700 5 -- op tracker -- seq: 47107, time: 2014-11-08 18:15:20.528058, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4711> 2014-11-08 18:15:20.528108 7fc5a152e700 5 -- op tracker -- seq: 47107, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4710> 2014-11-08 18:15:20.528150 7fc5b0316700 5 -- op tracker -- seq: 47107, time: 2014-11-08 18:15:20.528150, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4709> 2014-11-08 18:15:20.528159 7fc5b0316700 5 -- op tracker -- seq: 47107, time: 2014-11-08 18:15:20.528159, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4708> 2014-11-08 18:15:20.528206 7fc5b0316700 5 -- op tracker -- seq: 47107, time: 2014-11-08 18:15:20.528206, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb f5a20db5/1000002aa61.00000033/head//4) -4707> 2014-11-08 18:15:20.545445 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(817a8b16/1000002b105.000000a3/head//4)]) v2 -- ?+0 0x1d3af200 con 0x1c5c0580 -4706> 2014-11-08 18:15:20.547662 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8589 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) v3 ==== 757+0+0 (1704050608 0 0) 0x1e807500 con 0x1c5c0580 -4705> 2014-11-08 18:15:20.547692 7fc5a152e700 5 -- op tracker -- seq: 47108, time: 2014-11-08 18:15:20.547597, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4704> 2014-11-08 18:15:20.547703 7fc5a152e700 5 -- op tracker -- seq: 47108, time: 2014-11-08 18:15:20.547600, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4703> 2014-11-08 18:15:20.547708 7fc5a152e700 5 -- op tracker -- seq: 47108, time: 2014-11-08 18:15:20.547656, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4702> 2014-11-08 18:15:20.547713 7fc5a152e700 5 -- op tracker -- seq: 47108, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4701> 2014-11-08 18:15:20.547750 7fc5ad310700 5 -- op tracker -- seq: 47108, time: 2014-11-08 18:15:20.547750, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4700> 2014-11-08 18:15:20.547760 7fc5ad310700 5 -- op tracker -- seq: 47108, time: 2014-11-08 18:15:20.547759, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4699> 2014-11-08 18:15:20.547803 7fc5ad310700 5 -- op tracker -- seq: 47108, time: 2014-11-08 18:15:20.547803, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 817a8b16/1000002b105.000000a3/head//4) -4698> 2014-11-08 18:15:20.683900 7fc5c362f700 5 osd.9 5773 tick -4697> 2014-11-08 18:15:21.308076 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 507 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:21.305909) v2 ==== 47+0+0 (2643427404 0 0) 0x1e6acec0 con 0x1eb9d800 -4696> 2014-11-08 18:15:21.308103 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.305909) v2 -- ?+0 0x1e6ad0a0 con 0x1eb9d800 -4695> 2014-11-08 18:15:21.308710 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 507 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:21.305909) v2 ==== 47+0+0 (2643427404 0 0) 0x1e6aeae0 con 0x1eb9c8e0 -4694> 2014-11-08 18:15:21.308723 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.305909) v2 -- ?+0 0x1e6aecc0 con 0x1eb9c8e0 -4693> 2014-11-08 18:15:21.433052 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8590 ==== MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b240db5/ 10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8505913+0+0 (1667409613 0 0) 0x1d3af200 con 0x1c5c0580 -4692> 2014-11-08 18:15:21.433099 7fc5a152e700 5 -- op tracker -- seq: 47109, time: 2014-11-08 18:15:21.282076, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 7b240db5/10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4691> 2014-11-08 18:15:21.433122 7fc5a152e700 5 -- op tracker -- seq: 47109, time: 2014-11-08 18:15:21.282081, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b240db5/ 10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4690> 2014-11-08 18:15:21.433142 7fc5a152e700 5 -- op tracker -- seq: 47109, time: 2014-11-08 18:15:21.433042, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b240db5/ 10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4689> 2014-11-08 18:15:21.433161 7fc5a152e700 5 -- op tracker -- seq: 47109, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b240db5/ 10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4688> 2014-11-08 18:15:21.433215 7fc5adb11700 5 -- op tracker -- seq: 47109, time: 2014-11-08 18:15:21.433215, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b240db5/ 10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4687> 2014-11-08 18:15:21.433402 7fc5adb11700 5 -- op tracker -- seq: 47109, time: 2014-11-08 18:15:21.433402, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(19530db5/1000002002e.00000000/head//4, version: 3033'3523, data_included: [0~112640], data_size: 112640, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(19530db5/1000002002e.00000000/head//4@3033'3523, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:225280, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e930db5/1000002a777.000000a2/head//4, version: 3458'5361, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e930db5/1000002a777.000000a2/head//4@3458'5361, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(11040db5/1000001b4ed.00000608/head//4, version: 3033'3148, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11040db5/1000001b4ed.00000608/head//4@3033'3148, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7b240db5/10000019b1e.0000038f/head//4, version: 397'1197, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7b240db5/ 10000019b1e.0000038f/head//4@397'1197, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(45340db5/10000019b35.000007b5/head//4, version: 397'1227, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(45340db5/10000019b35.000007b5/head//4@397'1227, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4686> 2014-11-08 18:15:21.567779 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239ed280 con 0x1c5c0580 -4685> 2014-11-08 18:15:21.567815 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239eba20 con 0x1c5c0580 -4684> 2014-11-08 18:15:21.567822 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239ee360 con 0x1c5c0580 -4683> 2014-11-08 18:15:21.567827 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x239e81e0 con 0x1c5c0580 -4682> 2014-11-08 18:15:21.567833 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2515f620 con 0x1c5c0580 -4681> 2014-11-08 18:15:21.684006 7fc5c362f700 5 osd.9 5773 tick -4680> 2014-11-08 18:15:21.758548 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -4679> 2014-11-08 18:15:21.758576 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x24859e00 con 0x1cae2940 -4678> 2014-11-08 18:15:21.758601 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1eac8780 con 0x1cae2680 -4677> 2014-11-08 18:15:21.758612 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x22994740 con 0x1c5c02c0 -4676> 2014-11-08 18:15:21.758621 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x2583c1a0 con 0x1cae0160 -4675> 2014-11-08 18:15:21.758630 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x259f4740 con 0x1cae0f20 -4674> 2014-11-08 18:15:21.758639 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x2485d820 con 0x1cae0c60 -4673> 2014-11-08 18:15:21.758648 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1eacb480 con 0x1cae23c0 -4672> 2014-11-08 18:15:21.758657 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1e535640 con 0x1cae56a0 -4671> 2014-11-08 18:15:21.758667 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1e6abfc0 con 0x1cbf9ce0 -4670> 2014-11-08 18:15:21.758678 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x2285c560 con 0x1d603020 -4669> 2014-11-08 18:15:21.758698 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1c835280 con 0x1dbd9b80 -4668> 2014-11-08 18:15:21.758708 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x25ed7440 con 0x1d235d80 -4667> 2014-11-08 18:15:21.758718 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x24959a40 con 0x1cae1fa0 -4666> 2014-11-08 18:15:21.758728 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1d5e2d00 con 0x1cae1ce0 -4665> 2014-11-08 18:15:21.758738 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x23685dc0 con 0x1cae1340 -4664> 2014-11-08 18:15:21.758748 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x2127cce0 con 0x1cae1080 -4663> 2014-11-08 18:15:21.758758 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1d1b0000 con 0x1cae1a20 -4662> 2014-11-08 18:15:21.758768 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:21.758573) v2 -- ?+0 0x1d0423a0 con 0x1cae2c00 -4661> 2014-11-08 18:15:21.759591 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x22990d20 con 0x1cae56a0 -4660> 2014-11-08 18:15:21.759617 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x23921c20 con 0x1cae0f20 -4659> 2014-11-08 18:15:21.759627 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x23926ea0 con 0x1cae0c60 -4658> 2014-11-08 18:15:21.759635 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1eacaee0 con 0x1cae1ce0 -4657> 2014-11-08 18:15:21.759645 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x23681e00 con 0x1cae2940 -4656> 2014-11-08 18:15:21.759653 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1d5e10e0 con 0x1cae2680 -4655> 2014-11-08 18:15:21.759662 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x22e983c0 con 0x1cae23c0 -4654> 2014-11-08 18:15:21.759716 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x2285c1a0 con 0x1cae0160 -4653> 2014-11-08 18:15:21.759763 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 326 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1eae5460 con 0x1d235d80 -4652> 2014-11-08 18:15:21.759774 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1eac9860 con 0x1cae1fa0 -4651> 2014-11-08 18:15:21.759784 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 326 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1ddcd0a0 con 0x1dbd9b80 -4650> 2014-11-08 18:15:21.759903 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1c831c20 con 0x1c5c02c0 -4649> 2014-11-08 18:15:21.760006 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x2127c920 con 0x1cae2c00 -4648> 2014-11-08 18:15:21.760019 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x22578000 con 0x1cae1a20 -4647> 2014-11-08 18:15:21.762391 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x25fd7440 con 0x1cae1340 -4646> 2014-11-08 18:15:21.762412 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3739 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:21.758573) v2 ==== 47+0+0 (703348470 0 0) 0x1d043480 con 0x1cae1080 -4645> 2014-11-08 18:15:22.256577 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3716 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.254194) v2 ==== 47+0+0 (1389254319 0 0) 0x24958f00 con 0x1c9f8580 -4644> 2014-11-08 18:15:22.256619 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.254194) v2 -- ?+0 0x1e6acec0 con 0x1c9f8580 -4643> 2014-11-08 18:15:22.256649 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3716 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.254194) v2 ==== 47+0+0 (1389254319 0 0) 0x1eae1c20 con 0x1c8b7d20 -4642> 2014-11-08 18:15:22.256658 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.254194) v2 -- ?+0 0x1e6aeae0 con 0x1c8b7d20 -4641> 2014-11-08 18:15:22.326711 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3749 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.325935) v2 ==== 47+0+0 (311673431 0 0) 0x2234f260 con 0x1cce3180 -4640> 2014-11-08 18:15:22.326739 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.325935) v2 -- ?+0 0x24958f00 con 0x1cce3180 -4639> 2014-11-08 18:15:22.326892 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3749 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.325935) v2 ==== 47+0+0 (311673431 0 0) 0x2127c380 con 0x1cce3700 -4638> 2014-11-08 18:15:22.326910 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.325935) v2 -- ?+0 0x1eae1c20 con 0x1cce3700 -4637> 2014-11-08 18:15:22.515393 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3685 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.512456) v2 ==== 47+0+0 (3480736662 0 0) 0x234e1e00 con 0x1cce3440 -4636> 2014-11-08 18:15:22.515427 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.512456) v2 -- ?+0 0x2127c380 con 0x1cce3440 -4635> 2014-11-08 18:15:22.515467 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3685 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.512456) v2 ==== 47+0+0 (3480736662 0 0) 0x1caab480 con 0x1cce2ec0 -4634> 2014-11-08 18:15:22.515478 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.512456) v2 -- ?+0 0x2234f260 con 0x1cce2ec0 -4633> 2014-11-08 18:15:22.651078 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8591 ==== MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (3211266901 0 0) 0x2393f400 con 0x1c5c0580 -4632> 2014-11-08 18:15:22.651133 7fc5a152e700 5 -- op tracker -- seq: 47110, time: 2014-11-08 18:15:22.521907, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4631> 2014-11-08 18:15:22.651153 7fc5a152e700 5 -- op tracker -- seq: 47110, time: 2014-11-08 18:15:22.521911, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4630> 2014-11-08 18:15:22.651167 7fc5a152e700 5 -- op tracker -- seq: 47110, time: 2014-11-08 18:15:22.651045, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4629> 2014-11-08 18:15:22.651180 7fc5a152e700 5 -- op tracker -- seq: 47110, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4628> 2014-11-08 18:15:22.651226 7fc5afb15700 5 -- op tracker -- seq: 47110, time: 2014-11-08 18:15:22.651226, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4627> 2014-11-08 18:15:22.651354 7fc5afb15700 5 -- op tracker -- seq: 47110, time: 2014-11-08 18:15:22.651353, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(d81b8b16/1000001b506.00000091/head//4, version: 3033'3196, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d81b8b16/1000001b506.00000091/head//4@3033'3196, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e7cb8b16/1000001a1dd.0000023b/head//4, version: 397'1787, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e7cb8b16/1000001a1dd.0000023b/head//4@397'1787, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(bdec8b16/10000028733.00000180/head//4, version: 3033'3957, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(bdec8b16/10000028733.00000180/head//4@3033'3957, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4626> 2014-11-08 18:15:22.687271 7fc5c362f700 5 osd.9 5773 tick -4625> 2014-11-08 18:15:22.757040 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3717 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.754823) v2 ==== 47+0+0 (2630310758 0 0) 0x1eae7da0 con 0x1c8b7d20 -4624> 2014-11-08 18:15:22.757068 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.754823) v2 -- ?+0 0x234e1e00 con 0x1c8b7d20 -4623> 2014-11-08 18:15:22.757088 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3717 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:22.754823) v2 ==== 47+0+0 (2630310758 0 0) 0x25f97080 con 0x1c9f8580 -4622> 2014-11-08 18:15:22.757098 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:22.754823) v2 -- ?+0 0x1caab480 con 0x1c9f8580 -4621> 2014-11-08 18:15:22.849298 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25158960 con 0x1c5c0580 -4620> 2014-11-08 18:15:22.849341 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2515b480 con 0x1c5c0580 -4619> 2014-11-08 18:15:22.849348 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495ba20 con 0x1c5c0580 -4618> 2014-11-08 18:15:23.257439 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3718 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:23.255524) v2 ==== 47+0+0 (4098212731 0 0) 0x25f923a0 con 0x1c9f8580 -4617> 2014-11-08 18:15:23.257484 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:23.255524) v2 -- ?+0 0x25f97080 con 0x1c9f8580 -4616> 2014-11-08 18:15:23.257965 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3718 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:23.255524) v2 ==== 47+0+0 (4098212731 0 0) 0x1eae6ae0 con 0x1c8b7d20 -4615> 2014-11-08 18:15:23.257980 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:23.255524) v2 -- ?+0 0x1eae7da0 con 0x1c8b7d20 -4614> 2014-11-08 18:15:23.287786 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2264 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) v2 ==== 175+0+0 (1898050382 0 0) 0x229b9200 con 0x1e50aec0 -4613> 2014-11-08 18:15:23.287824 7fc5a56ef700 5 -- op tracker -- seq: 47111, time: 2014-11-08 18:15:23.287649, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -4612> 2014-11-08 18:15:23.287832 7fc5a56ef700 5 -- op tracker -- seq: 47111, time: 2014-11-08 18:15:23.287652, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -4611> 2014-11-08 18:15:23.287836 7fc5a56ef700 5 -- op tracker -- seq: 47111, time: 2014-11-08 18:15:23.287776, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -4610> 2014-11-08 18:15:23.287839 7fc5a56ef700 5 -- op tracker -- seq: 47111, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -4609> 2014-11-08 18:15:23.287955 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2265 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) v2 ==== 370+0+0 (326631787 0 0) 0x1d3a8000 con 0x1e50aec0 -4608> 2014-11-08 18:15:23.287982 7fc5a56ef700 5 -- op tracker -- seq: 47112, time: 2014-11-08 18:15:23.287928, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) -4607> 2014-11-08 18:15:23.287987 7fc5a56ef700 5 -- op tracker -- seq: 47112, time: 2014-11-08 18:15:23.287928, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) -4606> 2014-11-08 18:15:23.287992 7fc5a56ef700 5 -- op tracker -- seq: 47112, time: 2014-11-08 18:15:23.287954, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) -4605> 2014-11-08 18:15:23.287996 7fc5a56ef700 5 -- op tracker -- seq: 47112, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) -4604> 2014-11-08 18:15:23.288040 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2266 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) v2 ==== 240+0+0 (1162236462 0 0) 0x229b8800 con 0x1e50aec0 -4603> 2014-11-08 18:15:23.288049 7fc5a56ef700 5 -- op tracker -- seq: 47113, time: 2014-11-08 18:15:23.288013, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) -4602> 2014-11-08 18:15:23.288054 7fc5a56ef700 5 -- op tracker -- seq: 47113, time: 2014-11-08 18:15:23.288014, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) -4601> 2014-11-08 18:15:23.288057 7fc5a56ef700 5 -- op tracker -- seq: 47113, time: 2014-11-08 18:15:23.288039, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) -4600> 2014-11-08 18:15:23.288061 7fc5a56ef700 5 -- op tracker -- seq: 47113, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) -4599> 2014-11-08 18:15:23.288092 7fc5ad310700 5 -- op tracker -- seq: 47113, time: 2014-11-08 18:15:23.288092, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) -4598> 2014-11-08 18:15:23.288096 7fc5ac30e700 5 -- op tracker -- seq: 47111, time: 2014-11-08 18:15:23.288096, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -4597> 2014-11-08 18:15:23.288201 7fc5ac30e700 5 -- op tracker -- seq: 47111, time: 2014-11-08 18:15:23.288201, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(bd4edd13/1000002bc0a.0000001c/head//4),PushReplyOp(7e7edd13/1000002b5ea.000004c4/head//4)]) -4596> 2014-11-08 18:15:23.288208 7fc5ad310700 5 -- op tracker -- seq: 47113, time: 2014-11-08 18:15:23.288208, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(31d2e484/1000002abe7.0000000b/head//4),PushReplyOp(6a43e484/100000297d3.0000005e/head//4),PushReplyOp(ecb3e484/1000001a2bb.00000088/head//4)]) -4595> 2014-11-08 18:15:23.288216 7fc5ac30e700 5 -- op tracker -- seq: 47112, time: 2014-11-08 18:15:23.288215, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) -4594> 2014-11-08 18:15:23.288332 7fc5ac30e700 5 -- op tracker -- seq: 47112, time: 2014-11-08 18:15:23.288332, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(44260518/1000002ad5b.00000044/head//4),PushReplyOp(75f60518/1000001a0eb.00000616/head//4),PushReplyOp(cac70518/10000016074.00000000/head//4),PushReplyOp(9e080518/1000001a2e2.00000011/head//4),PushReplyOp(22c80518/1000002b9f1.0000000d/head//4)]) -4593> 2014-11-08 18:15:23.334320 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2267 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) v3 ==== 757+0+0 (415516961 0 0) 0x23cea880 con 0x1e50aec0 -4592> 2014-11-08 18:15:23.334350 7fc5a56ef700 5 -- op tracker -- seq: 47114, time: 2014-11-08 18:15:23.334219, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4591> 2014-11-08 18:15:23.334357 7fc5a56ef700 5 -- op tracker -- seq: 47114, time: 2014-11-08 18:15:23.334221, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4590> 2014-11-08 18:15:23.334360 7fc5a56ef700 5 -- op tracker -- seq: 47114, time: 2014-11-08 18:15:23.334314, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4589> 2014-11-08 18:15:23.334363 7fc5a56ef700 5 -- op tracker -- seq: 47114, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4588> 2014-11-08 18:15:23.334400 7fc5af314700 5 -- op tracker -- seq: 47114, time: 2014-11-08 18:15:23.334399, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4587> 2014-11-08 18:15:23.334414 7fc5af314700 5 -- op tracker -- seq: 47114, time: 2014-11-08 18:15:23.334414, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4586> 2014-11-08 18:15:23.334459 7fc5af314700 5 -- op tracker -- seq: 47114, time: 2014-11-08 18:15:23.334459, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb e36f90b7/10000019aff.000000d7/head//4) -4585> 2014-11-08 18:15:23.687376 7fc5c362f700 5 osd.9 5773 tick -4584> 2014-11-08 18:15:23.941559 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=fa1c0518/1000001a653.00000126/head//4,6e5c0518/100000283b6.00000047/head//4,feac0518/1000002bdbf.0000002f/head//4,c5bc0518/1000001984b.00000002/head//4,a95d0518/1000001a247.000000ba/head//4)) v1 -- ?+0 0x1ea7ad00 con 0x1c5c0580 -4583> 2014-11-08 18:15:23.941601 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7d340 con 0x42c86e0 -4582> 2014-11-08 18:15:23.941619 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is c36d0518/1000002c371.00000000/head//4 -4581> 2014-11-08 18:15:23.941647 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb 22c80518/1000002b9f1.0000000d/head//4) v3 -- ?+0 0x23ceb600 con 0x1e50aec0 -4580> 2014-11-08 18:15:23.941773 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7d340 con 0x42c86e0 -4579> 2014-11-08 18:15:23.941791 7fc5b9328700 5 -- op tracker -- seq: 47115, time: 2014-11-08 18:15:23.941612, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) -4578> 2014-11-08 18:15:23.941803 7fc5b9328700 5 -- op tracker -- seq: 47115, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) -4577> 2014-11-08 18:15:23.941809 7fc5b9328700 5 -- op tracker -- seq: 47115, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) -4576> 2014-11-08 18:15:23.941814 7fc5b9328700 5 -- op tracker -- seq: 47115, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) -4575> 2014-11-08 18:15:23.941849 7fc5ac30e700 5 -- op tracker -- seq: 47115, time: 2014-11-08 18:15:23.941849, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) -4574> 2014-11-08 18:15:24.002681 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 324 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:23.999529) v2 ==== 47+0+0 (1521907938 0 0) 0x239ea580 con 0x1eb9c780 -4573> 2014-11-08 18:15:24.002725 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:23.999529) v2 -- ?+0 0x1eae6ae0 con 0x1eb9c780 -4572> 2014-11-08 18:15:24.002750 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 324 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:23.999529) v2 ==== 47+0+0 (1521907938 0 0) 0x25fc4560 con 0x1eb9d6a0 -4571> 2014-11-08 18:15:24.002759 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:23.999529) v2 -- ?+0 0x25f923a0 con 0x1eb9d6a0 -4570> 2014-11-08 18:15:24.083659 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=1c44e484/1000002abd8.00000035/head//4,e765e484/10000029782.00000095/head//4,2d27e484/1000002b0f7.0000000f/head//4,e3c8e484/1000001b255.00000009/head//4,b009e484/1000001a70f.0000005f/head//4)) v1 -- ?+0 0x1ea7c5c0 con 0x1c5c0580 -4569> 2014-11-08 18:15:24.083718 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x1ea7c380 con 0x42c86e0 -4568> 2014-11-08 18:15:24.083736 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is c149e484/1000002a5cd.0000007d/head//4 -4567> 2014-11-08 18:15:24.083766 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb ecb3e484/1000001a2bb.00000088/head//4) v3 -- ?+0 0x213f6780 con 0x1e50aec0 -4566> 2014-11-08 18:15:24.083801 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x1ea7c380 con 0x42c86e0 -4565> 2014-11-08 18:15:24.083819 7fc5b9328700 5 -- op tracker -- seq: 47116, time: 2014-11-08 18:15:24.083729, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) -4564> 2014-11-08 18:15:24.083831 7fc5b9328700 5 -- op tracker -- seq: 47116, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) -4563> 2014-11-08 18:15:24.083836 7fc5b9328700 5 -- op tracker -- seq: 47116, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) -4562> 2014-11-08 18:15:24.083841 7fc5b9328700 5 -- op tracker -- seq: 47116, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) -4561> 2014-11-08 18:15:24.083977 7fc5afb15700 5 -- op tracker -- seq: 47116, time: 2014-11-08 18:15:24.083977, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) -4560> 2014-11-08 18:15:24.209599 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 508 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:24.206625) v2 ==== 47+0+0 (100237524 0 0) 0x1e6ae900 con 0x1eb9c8e0 -4559> 2014-11-08 18:15:24.209625 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:24.206625) v2 -- ?+0 0x239ea580 con 0x1eb9c8e0 -4558> 2014-11-08 18:15:24.209708 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 508 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:24.206625) v2 ==== 47+0+0 (100237524 0 0) 0x1e6acce0 con 0x1eb9d800 -4557> 2014-11-08 18:15:24.209718 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:24.206625) v2 -- ?+0 0x25fc4560 con 0x1eb9d800 -4556> 2014-11-08 18:15:24.521074 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8592 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) v1 ==== 10488228+0+0 (3815506877 0 0) 0x21a49680 con 0x1c5c0580 -4555> 2014-11-08 18:15:24.521110 7fc5a152e700 5 -- op tracker -- seq: 47117, time: 2014-11-08 18:15:24.391514, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) -4554> 2014-11-08 18:15:24.521119 7fc5a152e700 5 -- op tracker -- seq: 47117, time: 2014-11-08 18:15:24.391519, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) -4553> 2014-11-08 18:15:24.521121 7fc5a152e700 5 -- op tracker -- seq: 47117, time: 2014-11-08 18:15:24.521062, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) -4552> 2014-11-08 18:15:24.521125 7fc5a152e700 5 -- op tracker -- seq: 47117, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) -4551> 2014-11-08 18:15:24.527745 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) v1 -- ?+0 0x224bc380 con 0x42c86e0 -4550> 2014-11-08 18:15:24.527783 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224bc380 con 0x42c86e0 -4549> 2014-11-08 18:15:24.527794 7fc5b9328700 5 -- op tracker -- seq: 47118, time: 2014-11-08 18:15:24.527771, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) -4548> 2014-11-08 18:15:24.527802 7fc5b9328700 5 -- op tracker -- seq: 47118, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) -4547> 2014-11-08 18:15:24.527805 7fc5b9328700 5 -- op tracker -- seq: 47118, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) -4546> 2014-11-08 18:15:24.527829 7fc5b9328700 5 -- op tracker -- seq: 47118, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) -4545> 2014-11-08 18:15:24.527849 7fc5ac30e700 5 -- op tracker -- seq: 47115, time: 2014-11-08 18:15:24.527849, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4421, to_read={fa1c0518/1000001a653.00000126/head//4=0,4194304,6e5c0518/100000283b6.00000047/head//4=0,4194304,feac0518/1000002bdbf.0000002f/head//4=0,4194304,c5bc0518/1000001984b.00000002/head//4=0,4194304,a95d0518/1000001a247.000000ba/head//4=0,4194304}, attrs_to_read=)) -4544> 2014-11-08 18:15:24.527858 7fc5aeb13700 5 -- op tracker -- seq: 47117, time: 2014-11-08 18:15:24.527858, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) -4543> 2014-11-08 18:15:24.527914 7fc5aeb13700 5 -- op tracker -- seq: 47117, time: 2014-11-08 18:15:24.527914, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=5)) -4542> 2014-11-08 18:15:24.527934 7fc5ac30e700 5 -- op tracker -- seq: 47118, time: 2014-11-08 18:15:24.527934, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) -4541> 2014-11-08 18:15:24.547593 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12063 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) v1 ==== 377+0+0 (3510138454 0 0) 0x1ce67080 con 0x1c5c2680 -4540> 2014-11-08 18:15:24.547618 7fc5a6cfe700 5 -- op tracker -- seq: 47119, time: 2014-11-08 18:15:24.547525, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) -4539> 2014-11-08 18:15:24.547625 7fc5a6cfe700 5 -- op tracker -- seq: 47119, time: 2014-11-08 18:15:24.547528, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) -4538> 2014-11-08 18:15:24.547631 7fc5a6cfe700 5 -- op tracker -- seq: 47119, time: 2014-11-08 18:15:24.547587, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) -4537> 2014-11-08 18:15:24.547636 7fc5a6cfe700 5 -- op tracker -- seq: 47119, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) -4536> 2014-11-08 18:15:24.555718 7fc5aeb13700 5 -- op tracker -- seq: 47119, time: 2014-11-08 18:15:24.555718, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) -4535> 2014-11-08 18:15:24.561819 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(fa1c0518/1000001a653.00000126/head//4, version: 397'2152, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fa1c0518/1000001a653.00000126/head//4@397'2152, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e5c0518/100000283b6.00000047/head//4, version: 3033'3752, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e5c0518/100000283b6.00000047/head//4@3033'3752, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(feac0518/1000002bdbf.0000002f/head//4, version: 4864'8346, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(feac0518/1000002bdbf.0000002f/head//4@4864'8346, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c5bc0518/1000001984b.00000002/head//4, version: 397'931, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c5bc0518/1000001984b.00000002/head/ /4@397'931, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a95d0518/1000001a247.000000ba/head//4, version: 397'2002, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a95d0518/1000001a247.000000ba/head//4@397'2002, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3a8000 con 0x1e50aec0 -4534> 2014-11-08 18:15:24.568472 7fc5ac30e700 5 -- op tracker -- seq: 47118, time: 2014-11-08 18:15:24.568472, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4421, attrs_read=0)) -4533> 2014-11-08 18:15:24.581941 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) v1 -- ?+0 0x21a49e00 con 0x42c86e0 -4532> 2014-11-08 18:15:24.581991 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21a49e00 con 0x42c86e0 -4531> 2014-11-08 18:15:24.581991 7fc5afb15700 5 -- op tracker -- seq: 47116, time: 2014-11-08 18:15:24.581991, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4422, to_read={1c44e484/1000002abd8.00000035/head//4=0,4194304,e765e484/10000029782.00000095/head//4=0,4194304,2d27e484/1000002b0f7.0000000f/head//4=0,4194304,e3c8e484/1000001b255.00000009/head//4=0,4194304,b009e484/1000001a70f.0000005f/head//4=0,4194304}, attrs_to_read=)) -4530> 2014-11-08 18:15:24.582008 7fc5b9328700 5 -- op tracker -- seq: 47120, time: 2014-11-08 18:15:24.581979, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) -4529> 2014-11-08 18:15:24.582012 7fc5b9328700 5 -- op tracker -- seq: 47120, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) -4528> 2014-11-08 18:15:24.582014 7fc5b9328700 5 -- op tracker -- seq: 47120, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) -4527> 2014-11-08 18:15:24.582016 7fc5b9328700 5 -- op tracker -- seq: 47120, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) -4526> 2014-11-08 18:15:24.582073 7fc5ad310700 5 -- op tracker -- seq: 47120, time: 2014-11-08 18:15:24.582073, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) -4525> 2014-11-08 18:15:24.582116 7fc5ad310700 5 -- op tracker -- seq: 47120, time: 2014-11-08 18:15:24.582116, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=0)) -4524> 2014-11-08 18:15:24.687461 7fc5c362f700 5 osd.9 5773 tick -4523> 2014-11-08 18:15:24.707755 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8593 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) v1 ==== 10488220+0+0 (809677455 0 0) 0x21a4eb80 con 0x1c5c0580 -4522> 2014-11-08 18:15:24.711928 7fc5a152e700 5 -- op tracker -- seq: 47121, time: 2014-11-08 18:15:24.521185, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) -4521> 2014-11-08 18:15:24.711952 7fc5a152e700 5 -- op tracker -- seq: 47121, time: 2014-11-08 18:15:24.521186, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) -4520> 2014-11-08 18:15:24.711962 7fc5a152e700 5 -- op tracker -- seq: 47121, time: 2014-11-08 18:15:24.707742, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) -4519> 2014-11-08 18:15:24.711971 7fc5a152e700 5 -- op tracker -- seq: 47121, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) -4518> 2014-11-08 18:15:24.712015 7fc5afb15700 5 -- op tracker -- seq: 47121, time: 2014-11-08 18:15:24.712015, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) -4517> 2014-11-08 18:15:24.740427 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(1c44e484/1000002abd8.00000035/head//4, version: 3468'5859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1c44e484/1000002abd8.00000035/head//4@3468'5859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e765e484/10000029782.00000095/head//4, version: 3033'4306, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e765e484/10000029782.00000095/head//4@3033'4306, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2d27e484/1000002b0f7.0000000f/head//4, version: 4157'7822, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2d27e484/1000002b0f7.0000000f/head//4@4157'7822, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e3c8e484/1000001b255.00000009/head//4, version: 2829'2628, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e3c8e484/ 1000001b255.00000009/head//4@2829'2628, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b009e484/1000001a70f.0000005f/head//4, version: 397'2055, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b009e484/1000001a70f.0000005f/head//4@397'2055, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393f400 con 0x1e50aec0 -4516> 2014-11-08 18:15:24.756729 7fc5afb15700 5 -- op tracker -- seq: 47121, time: 2014-11-08 18:15:24.756728, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4422, attrs_read=5)) -4515> 2014-11-08 18:15:24.805956 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2268 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x4532940 con 0x1e50aec0 -4514> 2014-11-08 18:15:24.806006 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.805608, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4513> 2014-11-08 18:15:24.806016 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.805612, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4512> 2014-11-08 18:15:24.806019 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.805878, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4511> 2014-11-08 18:15:24.806022 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.805998, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4510> 2014-11-08 18:15:24.806026 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.806026, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4509> 2014-11-08 18:15:24.806037 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.806037, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4508> 2014-11-08 18:15:24.806043 7fc5b9b29700 5 -- op tracker -- seq: 47122, time: 2014-11-08 18:15:24.806043, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4507> 2014-11-08 18:15:24.806059 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2269 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x4531a40 con 0x1e50aec0 -4506> 2014-11-08 18:15:24.806065 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.805896, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4505> 2014-11-08 18:15:24.806069 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.805900, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4504> 2014-11-08 18:15:24.806071 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.805920, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4503> 2014-11-08 18:15:24.806074 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.806064, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4502> 2014-11-08 18:15:24.806077 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.806077, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4501> 2014-11-08 18:15:24.806080 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.806080, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4500> 2014-11-08 18:15:24.806084 7fc5b9b29700 5 -- op tracker -- seq: 47123, time: 2014-11-08 18:15:24.806084, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4499> 2014-11-08 18:15:24.806090 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2270 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x4535dc0 con 0x1e50aec0 -4498> 2014-11-08 18:15:24.806095 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.805925, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4497> 2014-11-08 18:15:24.806098 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.805925, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4496> 2014-11-08 18:15:24.806101 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.805944, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4495> 2014-11-08 18:15:24.806104 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.806095, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4494> 2014-11-08 18:15:24.806106 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.806106, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4493> 2014-11-08 18:15:24.806109 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.806109, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4492> 2014-11-08 18:15:24.806113 7fc5b9b29700 5 -- op tracker -- seq: 47124, time: 2014-11-08 18:15:24.806113, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4491> 2014-11-08 18:15:24.806328 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2271 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x4536180 con 0x1e50aec0 -4490> 2014-11-08 18:15:24.806340 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806292, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4489> 2014-11-08 18:15:24.806343 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806293, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4488> 2014-11-08 18:15:24.806346 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806318, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4487> 2014-11-08 18:15:24.806349 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806338, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4486> 2014-11-08 18:15:24.806352 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806352, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4485> 2014-11-08 18:15:24.806356 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806356, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4484> 2014-11-08 18:15:24.806359 7fc5b9b29700 5 -- op tracker -- seq: 47125, time: 2014-11-08 18:15:24.806359, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4483> 2014-11-08 18:15:24.806365 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2272 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x45321c0 con 0x1e50aec0 -4482> 2014-11-08 18:15:24.806371 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806326, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4481> 2014-11-08 18:15:24.806374 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806326, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4480> 2014-11-08 18:15:24.806377 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806345, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4479> 2014-11-08 18:15:24.806379 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806371, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4478> 2014-11-08 18:15:24.806382 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806382, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4477> 2014-11-08 18:15:24.806385 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806385, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4476> 2014-11-08 18:15:24.806377 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2273 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) v2 ==== 370+0+0 (3940718866 0 0) 0x1d3a8000 con 0x1e50aec0 -4475> 2014-11-08 18:15:24.806388 7fc5b9b29700 5 -- op tracker -- seq: 47126, time: 2014-11-08 18:15:24.806388, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4474> 2014-11-08 18:15:24.806391 7fc5a56ef700 5 -- op tracker -- seq: 47127, time: 2014-11-08 18:15:24.806350, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) -4473> 2014-11-08 18:15:24.806397 7fc5a56ef700 5 -- op tracker -- seq: 47127, time: 2014-11-08 18:15:24.806350, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) -4472> 2014-11-08 18:15:24.806401 7fc5a56ef700 5 -- op tracker -- seq: 47127, time: 2014-11-08 18:15:24.806377, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) -4471> 2014-11-08 18:15:24.806406 7fc5a56ef700 5 -- op tracker -- seq: 47127, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) -4470> 2014-11-08 18:15:24.806485 7fc5ac30e700 5 -- op tracker -- seq: 47127, time: 2014-11-08 18:15:24.806485, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) -4469> 2014-11-08 18:15:24.806632 7fc5ac30e700 5 -- op tracker -- seq: 47127, time: 2014-11-08 18:15:24.806632, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(fa1c0518/1000001a653.00000126/head//4),PushReplyOp(6e5c0518/100000283b6.00000047/head//4),PushReplyOp(feac0518/1000002bdbf.0000002f/head//4),PushReplyOp(c5bc0518/1000001984b.00000002/head//4),PushReplyOp(a95d0518/1000001a247.000000ba/head//4)]) -4468> 2014-11-08 18:15:24.867753 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=c36d0518/1000002c371.00000000/head//4)) v1 -- ?+0 0x1ea7f980 con 0x1c5c0580 -4467> 2014-11-08 18:15:24.867797 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2287e0c0 con 0x42c86e0 -4466> 2014-11-08 18:15:24.867809 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 596d0518/10000019e0c.0000020a/head//4 -4465> 2014-11-08 18:15:24.867854 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb a95d0518/1000001a247.000000ba/head//4) v3 -- ?+0 0x23cea400 con 0x1e50aec0 -4464> 2014-11-08 18:15:24.868135 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2287e0c0 con 0x42c86e0 -4463> 2014-11-08 18:15:24.868147 7fc5b9328700 5 -- op tracker -- seq: 47128, time: 2014-11-08 18:15:24.867804, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) -4462> 2014-11-08 18:15:24.868156 7fc5b9328700 5 -- op tracker -- seq: 47128, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) -4461> 2014-11-08 18:15:24.868159 7fc5b9328700 5 -- op tracker -- seq: 47128, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) -4460> 2014-11-08 18:15:24.868161 7fc5b9328700 5 -- op tracker -- seq: 47128, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) -4459> 2014-11-08 18:15:24.868194 7fc5ac30e700 5 -- op tracker -- seq: 47128, time: 2014-11-08 18:15:24.868194, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) -4458> 2014-11-08 18:15:24.868237 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) v1 -- ?+0 0x224bc380 con 0x42c86e0 -4457> 2014-11-08 18:15:24.868248 7fc5ac30e700 5 -- op tracker -- seq: 47128, time: 2014-11-08 18:15:24.868248, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4423, to_read={c36d0518/1000002c371.00000000/head//4=0,4194304}, attrs_to_read=)) -4456> 2014-11-08 18:15:24.868265 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224bc380 con 0x42c86e0 -4455> 2014-11-08 18:15:24.868272 7fc5b9328700 5 -- op tracker -- seq: 47129, time: 2014-11-08 18:15:24.868244, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) -4454> 2014-11-08 18:15:24.868275 7fc5b9328700 5 -- op tracker -- seq: 47129, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) -4453> 2014-11-08 18:15:24.868277 7fc5b9328700 5 -- op tracker -- seq: 47129, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) -4452> 2014-11-08 18:15:24.868278 7fc5b9328700 5 -- op tracker -- seq: 47129, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) -4451> 2014-11-08 18:15:24.868292 7fc5ac30e700 5 -- op tracker -- seq: 47129, time: 2014-11-08 18:15:24.868292, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) -4450> 2014-11-08 18:15:24.868307 7fc5ac30e700 5 -- op tracker -- seq: 47129, time: 2014-11-08 18:15:24.868307, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=0)) -4449> 2014-11-08 18:15:24.887141 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8594 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) v1 ==== 2934+0+0 (3276540555 0 0) 0x21a4b980 con 0x1c5c0580 -4448> 2014-11-08 18:15:24.887176 7fc5a152e700 5 -- op tracker -- seq: 47130, time: 2014-11-08 18:15:24.887034, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) -4447> 2014-11-08 18:15:24.887184 7fc5a152e700 5 -- op tracker -- seq: 47130, time: 2014-11-08 18:15:24.887037, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) -4446> 2014-11-08 18:15:24.887187 7fc5a152e700 5 -- op tracker -- seq: 47130, time: 2014-11-08 18:15:24.887132, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) -4445> 2014-11-08 18:15:24.887189 7fc5a152e700 5 -- op tracker -- seq: 47130, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) -4444> 2014-11-08 18:15:24.887230 7fc5ac30e700 5 -- op tracker -- seq: 47130, time: 2014-11-08 18:15:24.887230, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) -4443> 2014-11-08 18:15:24.887322 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(c36d0518/1000002c371.00000000/head//4, version: 5734'8400, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(c36d0518/1000002c371.00000000/head//4@5734'8400, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3a8000 con 0x1e50aec0 -4442> 2014-11-08 18:15:24.887358 7fc5ac30e700 5 -- op tracker -- seq: 47130, time: 2014-11-08 18:15:24.887358, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4423, attrs_read=1)) -4441> 2014-11-08 18:15:25.015271 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25195, attrs_read=0)) v1 -- ?+0 0x21a49680 con 0x1c5c2680 -4440> 2014-11-08 18:15:25.016709 7fc5aeb13700 5 -- op tracker -- seq: 47119, time: 2014-11-08 18:15:25.016708, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25195, to_read={e1214227/1000002b5b1.000000a7/head//4=0,4194304,c5224227/10000019b0c.00000052/head//4=0,4194304,97324227/1000001ac7c.0000004a/head//4=0,4194304,2d534227/1000001accc.00000024/head//4=0,4194304}, attrs_to_read=)) -4439> 2014-11-08 18:15:25.100664 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2274 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x4536720 con 0x1e50aec0 -4438> 2014-11-08 18:15:25.100715 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100562, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -4437> 2014-11-08 18:15:25.100723 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100566, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -4436> 2014-11-08 18:15:25.100725 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100639, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -4435> 2014-11-08 18:15:25.100728 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100710, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -4434> 2014-11-08 18:15:25.100731 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100731, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -4433> 2014-11-08 18:15:25.100739 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100739, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -4432> 2014-11-08 18:15:25.100744 7fc5b9b29700 5 -- op tracker -- seq: 47131, time: 2014-11-08 18:15:25.100744, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -4431> 2014-11-08 18:15:25.100836 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2275 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x4536ea0 con 0x1e50aec0 -4430> 2014-11-08 18:15:25.100846 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100802, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -4429> 2014-11-08 18:15:25.100849 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100802, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -4428> 2014-11-08 18:15:25.100852 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100825, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -4427> 2014-11-08 18:15:25.100854 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100845, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -4426> 2014-11-08 18:15:25.100857 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100857, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -4425> 2014-11-08 18:15:25.100861 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100861, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -4424> 2014-11-08 18:15:25.100864 7fc5b9b29700 5 -- op tracker -- seq: 47132, time: 2014-11-08 18:15:25.100864, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -4423> 2014-11-08 18:15:25.100912 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2276 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x4530b40 con 0x1e50aec0 -4422> 2014-11-08 18:15:25.100921 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100882, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -4421> 2014-11-08 18:15:25.100925 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100882, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -4420> 2014-11-08 18:15:25.100927 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100903, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -4419> 2014-11-08 18:15:25.100930 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100920, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -4418> 2014-11-08 18:15:25.100932 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100932, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -4417> 2014-11-08 18:15:25.100936 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100936, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -4416> 2014-11-08 18:15:25.100939 7fc5b9b29700 5 -- op tracker -- seq: 47133, time: 2014-11-08 18:15:25.100939, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -4415> 2014-11-08 18:15:25.100986 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2277 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x23d77440 con 0x1e50aec0 -4414> 2014-11-08 18:15:25.100995 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.100956, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -4413> 2014-11-08 18:15:25.100999 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.100956, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -4412> 2014-11-08 18:15:25.101022 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.100977, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -4411> 2014-11-08 18:15:25.101025 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.100994, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -4410> 2014-11-08 18:15:25.101028 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.101028, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -4409> 2014-11-08 18:15:25.101032 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.101032, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -4408> 2014-11-08 18:15:25.101035 7fc5b9b29700 5 -- op tracker -- seq: 47134, time: 2014-11-08 18:15:25.101035, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -4407> 2014-11-08 18:15:25.101082 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2278 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x23d705a0 con 0x1e50aec0 -4406> 2014-11-08 18:15:25.101091 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101053, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -4405> 2014-11-08 18:15:25.101095 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101053, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -4404> 2014-11-08 18:15:25.101097 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101073, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -4403> 2014-11-08 18:15:25.101099 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101090, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -4402> 2014-11-08 18:15:25.101102 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101102, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -4401> 2014-11-08 18:15:25.101106 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101106, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -4400> 2014-11-08 18:15:25.101110 7fc5b9b29700 5 -- op tracker -- seq: 47135, time: 2014-11-08 18:15:25.101110, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -4399> 2014-11-08 18:15:25.101154 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2279 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) v2 ==== 370+0+0 (1978562522 0 0) 0x1d3a8000 con 0x1e50aec0 -4398> 2014-11-08 18:15:25.101182 7fc5a56ef700 5 -- op tracker -- seq: 47136, time: 2014-11-08 18:15:25.101126, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) -4397> 2014-11-08 18:15:25.101203 7fc5a56ef700 5 -- op tracker -- seq: 47136, time: 2014-11-08 18:15:25.101127, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) -4396> 2014-11-08 18:15:25.101215 7fc5a56ef700 5 -- op tracker -- seq: 47136, time: 2014-11-08 18:15:25.101154, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) -4395> 2014-11-08 18:15:25.101226 7fc5a56ef700 5 -- op tracker -- seq: 47136, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) -4394> 2014-11-08 18:15:25.101270 7fc5ad310700 5 -- op tracker -- seq: 47136, time: 2014-11-08 18:15:25.101270, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) -4393> 2014-11-08 18:15:25.101432 7fc5ad310700 5 -- op tracker -- seq: 47136, time: 2014-11-08 18:15:25.101432, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(1c44e484/1000002abd8.00000035/head//4),PushReplyOp(e765e484/10000029782.00000095/head//4),PushReplyOp(2d27e484/1000002b0f7.0000000f/head//4),PushReplyOp(e3c8e484/1000001b255.00000009/head//4),PushReplyOp(b009e484/1000001a70f.0000005f/head//4)]) -4392> 2014-11-08 18:15:25.104550 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2280 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x211e4740 con 0x1e50aec0 -4391> 2014-11-08 18:15:25.104571 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104485, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4390> 2014-11-08 18:15:25.104576 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104487, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4389> 2014-11-08 18:15:25.104579 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104533, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4388> 2014-11-08 18:15:25.104574 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2281 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) v2 ==== 110+0+0 (338068180 0 0) 0x2393f400 con 0x1e50aec0 -4387> 2014-11-08 18:15:25.104582 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104568, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4386> 2014-11-08 18:15:25.104585 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104585, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4385> 2014-11-08 18:15:25.104585 7fc5a56ef700 5 -- op tracker -- seq: 47138, time: 2014-11-08 18:15:25.104546, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) -4384> 2014-11-08 18:15:25.104589 7fc5a56ef700 5 -- op tracker -- seq: 47138, time: 2014-11-08 18:15:25.104546, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) -4383> 2014-11-08 18:15:25.104590 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104590, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4382> 2014-11-08 18:15:25.104592 7fc5a56ef700 5 -- op tracker -- seq: 47138, time: 2014-11-08 18:15:25.104574, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) -4381> 2014-11-08 18:15:25.104595 7fc5a56ef700 5 -- op tracker -- seq: 47138, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) -4380> 2014-11-08 18:15:25.104596 7fc5b9b29700 5 -- op tracker -- seq: 47137, time: 2014-11-08 18:15:25.104595, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4379> 2014-11-08 18:15:25.104622 7fc5ac30e700 5 -- op tracker -- seq: 47138, time: 2014-11-08 18:15:25.104622, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) -4378> 2014-11-08 18:15:25.104708 7fc5ac30e700 5 -- op tracker -- seq: 47138, time: 2014-11-08 18:15:25.104708, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(c36d0518/1000002c371.00000000/head//4)]) -4377> 2014-11-08 18:15:25.127661 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(d81b8b16/1000001b506.00000091/head//4),PushReplyOp(e7cb8b16/1000001a1dd.0000023b/head//4),PushReplyOp(bdec8b16/10000028733.00000180/head//4)]) v2 -- ?+0 0x1d3ac200 con 0x1c5c0580 -4376> 2014-11-08 18:15:25.133627 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8595 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) v3 ==== 757+0+0 (4112648767 0 0) 0x1e806780 con 0x1c5c0580 -4375> 2014-11-08 18:15:25.133667 7fc5a152e700 5 -- op tracker -- seq: 47139, time: 2014-11-08 18:15:25.133510, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4374> 2014-11-08 18:15:25.133676 7fc5a152e700 5 -- op tracker -- seq: 47139, time: 2014-11-08 18:15:25.133514, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4373> 2014-11-08 18:15:25.133680 7fc5a152e700 5 -- op tracker -- seq: 47139, time: 2014-11-08 18:15:25.133615, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4372> 2014-11-08 18:15:25.133683 7fc5a152e700 5 -- op tracker -- seq: 47139, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4371> 2014-11-08 18:15:25.133748 7fc5afb15700 5 -- op tracker -- seq: 47139, time: 2014-11-08 18:15:25.133748, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4370> 2014-11-08 18:15:25.133757 7fc5afb15700 5 -- op tracker -- seq: 47139, time: 2014-11-08 18:15:25.133757, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4369> 2014-11-08 18:15:25.133809 7fc5afb15700 5 -- op tracker -- seq: 47139, time: 2014-11-08 18:15:25.133809, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb bdec8b16/10000028733.00000180/head//4) -4368> 2014-11-08 18:15:25.304128 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12064 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3286587554 0 0) 0x1ce60900 con 0x1c5c2680 -4367> 2014-11-08 18:15:25.304155 7fc5a6cfe700 5 -- op tracker -- seq: 47140, time: 2014-11-08 18:15:25.304050, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) -4366> 2014-11-08 18:15:25.304163 7fc5a6cfe700 5 -- op tracker -- seq: 47140, time: 2014-11-08 18:15:25.304053, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) -4365> 2014-11-08 18:15:25.304167 7fc5a6cfe700 5 -- op tracker -- seq: 47140, time: 2014-11-08 18:15:25.304121, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) -4364> 2014-11-08 18:15:25.304171 7fc5a6cfe700 5 -- op tracker -- seq: 47140, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) -4363> 2014-11-08 18:15:25.304211 7fc5aeb13700 5 -- op tracker -- seq: 47140, time: 2014-11-08 18:15:25.304211, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) -4362> 2014-11-08 18:15:25.305487 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12065 ==== MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) v1 ==== 298+0+0 (1766851583 0 0) 0x1ce62880 con 0x1c5c2680 -4361> 2014-11-08 18:15:25.305502 7fc5a6cfe700 5 -- op tracker -- seq: 47141, time: 2014-11-08 18:15:25.305430, event: header_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) -4360> 2014-11-08 18:15:25.305510 7fc5a6cfe700 5 -- op tracker -- seq: 47141, time: 2014-11-08 18:15:25.305432, event: throttled, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) -4359> 2014-11-08 18:15:25.305516 7fc5a6cfe700 5 -- op tracker -- seq: 47141, time: 2014-11-08 18:15:25.305483, event: all_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) -4358> 2014-11-08 18:15:25.305522 7fc5a6cfe700 5 -- op tracker -- seq: 47141, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) -4357> 2014-11-08 18:15:25.305546 7fc5ad310700 5 -- op tracker -- seq: 47141, time: 2014-11-08 18:15:25.305546, event: reached_pg, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) -4356> 2014-11-08 18:15:25.309750 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 509 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:25.307421) v2 ==== 47+0+0 (651365512 0 0) 0x1e6acb00 con 0x1eb9d800 -4355> 2014-11-08 18:15:25.309776 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:25.307421) v2 -- ?+0 0x1e6acce0 con 0x1eb9d800 -4354> 2014-11-08 18:15:25.309796 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 509 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:25.307421) v2 ==== 47+0+0 (651365512 0 0) 0x1e6ae720 con 0x1eb9c8e0 -4353> 2014-11-08 18:15:25.309807 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:25.307421) v2 -- ?+0 0x1e6ae900 con 0x1eb9c8e0 -4352> 2014-11-08 18:15:25.421126 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25196, attrs_read=0)) v1 -- ?+0 0x44a1b80 con 0x1c5c2680 -4351> 2014-11-08 18:15:25.421171 7fc5aeb13700 5 -- op tracker -- seq: 47140, time: 2014-11-08 18:15:25.421170, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25196, to_read={76e34227/1000002b1de.0000006e/head//4=0,4194304}, attrs_to_read=)) -4350> 2014-11-08 18:15:25.559161 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3719 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:25.556897) v2 ==== 47+0+0 (4030033566 0 0) 0x22d55dc0 con 0x1c8b7d20 -4349> 2014-11-08 18:15:25.559204 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:25.556897) v2 -- ?+0 0x1e6ae720 con 0x1c8b7d20 -4348> 2014-11-08 18:15:25.559234 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3719 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:25.556897) v2 ==== 47+0+0 (4030033566 0 0) 0x2515d820 con 0x1c9f8580 -4347> 2014-11-08 18:15:25.559246 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:25.556897) v2 -- ?+0 0x1e6acb00 con 0x1c9f8580 -4346> 2014-11-08 18:15:25.566840 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12066 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3865433648 0 0) 0x1ce63cc0 con 0x1c5c2680 -4345> 2014-11-08 18:15:25.566861 7fc5a6cfe700 5 -- op tracker -- seq: 47142, time: 2014-11-08 18:15:25.566785, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) -4344> 2014-11-08 18:15:25.566869 7fc5a6cfe700 5 -- op tracker -- seq: 47142, time: 2014-11-08 18:15:25.566788, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) -4343> 2014-11-08 18:15:25.566873 7fc5a6cfe700 5 -- op tracker -- seq: 47142, time: 2014-11-08 18:15:25.566835, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) -4342> 2014-11-08 18:15:25.566878 7fc5a6cfe700 5 -- op tracker -- seq: 47142, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) -4341> 2014-11-08 18:15:25.566907 7fc5ac30e700 5 -- op tracker -- seq: 47142, time: 2014-11-08 18:15:25.566907, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) -4340> 2014-11-08 18:15:25.608306 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25199, attrs_read=0)) v1 -- ?+0 0x21a4b980 con 0x1c5c2680 -4339> 2014-11-08 18:15:25.608361 7fc5ac30e700 5 -- op tracker -- seq: 47142, time: 2014-11-08 18:15:25.608360, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25199, to_read={58244227/1000002b1db.00000043/head//4=0,4194304}, attrs_to_read=)) -4338> 2014-11-08 18:15:25.687566 7fc5c362f700 5 osd.9 5773 tick -4337> 2014-11-08 18:15:25.967388 7fc5a1932700 2 -- 192.168.1.18:6814/4928 >> 192.168.1.17:6805/32391 pipe(0x247439c0 sd=48 :37890 s=2 pgs=2335 cs=1 l=1 c=0x1ec17d20).reader couldn't read tag, (0) Success -4336> 2014-11-08 18:15:25.967457 7fc5a1932700 2 -- 192.168.1.18:6814/4928 >> 192.168.1.17:6805/32391 pipe(0x247439c0 sd=48 :37890 s=2 pgs=2335 cs=1 l=1 c=0x1ec17d20).fault (0) Success -4335> 2014-11-08 18:15:25.967602 7fc5b431e700 1 osd.9.objecter ms_handle_reset on osd.8 -4334> 2014-11-08 18:15:25.967611 7fc5b431e700 1 -- 192.168.1.18:6814/4928 mark_down 0x1ec17d20 -- pipe dne -4333> 2014-11-08 18:15:26.017545 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3686 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.014616) v2 ==== 47+0+0 (3627528338 0 0) 0x1caadfa0 con 0x1cce2ec0 -4332> 2014-11-08 18:15:26.017730 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3686 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.014616) v2 ==== 47+0+0 (3627528338 0 0) 0x234e6ae0 con 0x1cce3440 -4331> 2014-11-08 18:15:25.687596 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:05.687593) -4330> 2014-11-08 18:15:26.044976 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -4329> 2014-11-08 18:15:26.045007 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- osd_failure(failed osd.4 192.168.1.16:6805/31236 for 21sec e5773 v5773) v3 -- ?+0 0x44a2f80 con 0x44ed6a0 -4328> 2014-11-08 18:15:26.045036 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -4327> 2014-11-08 18:15:26.045039 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- pg_stats(3 pgs tid 1331 v 0) v1 -- ?+0 0x248a8dc0 con 0x44ed6a0 -4326> 2014-11-08 18:15:26.045144 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.014616) v2 -- ?+0 0x2515d820 con 0x1cce2ec0 -4325> 2014-11-08 18:15:26.045226 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.014616) v2 -- ?+0 0x22d55dc0 con 0x1cce3440 -4324> 2014-11-08 18:15:26.049090 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2282 ==== MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 3252+0+0 (3082778688 0 0) 0x1d3af000 con 0x1e50aec0 -4323> 2014-11-08 18:15:26.049125 7fc5a56ef700 5 -- op tracker -- seq: 47143, time: 2014-11-08 18:15:26.046569, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4322> 2014-11-08 18:15:26.049135 7fc5a56ef700 5 -- op tracker -- seq: 47143, time: 2014-11-08 18:15:26.046571, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4321> 2014-11-08 18:15:26.049140 7fc5a56ef700 5 -- op tracker -- seq: 47143, time: 2014-11-08 18:15:26.049081, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4320> 2014-11-08 18:15:26.049146 7fc5a56ef700 5 -- op tracker -- seq: 47143, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4319> 2014-11-08 18:15:26.049189 7fc5acb0f700 5 -- op tracker -- seq: 47143, time: 2014-11-08 18:15:26.049189, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4318> 2014-11-08 18:15:26.049270 7fc5acb0f700 5 -- op tracker -- seq: 47143, time: 2014-11-08 18:15:26.049270, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(a99f90b7/1000001cfbf.00000000/head//4, version: 3033'3408, data_included: [0~2048], data_size: 2048, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(a99f90b7/1000001cfbf.00000000/head//4@3033'3408, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4096, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4317> 2014-11-08 18:15:26.105283 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x249590e0 con 0x1e50aec0 -4316> 2014-11-08 18:15:26.136562 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.bbs0 5773 ECSubReadReply(tid=25197, attrs_read=0)) v1 -- ?+0 0x21a49e00 con 0x1c5c2680 -4315> 2014-11-08 18:15:26.136611 7fc5ad310700 5 -- op tracker -- seq: 47141, time: 2014-11-08 18:15:26.136611, event: done, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25197, to_read={5e02a0bb/1000002bae2.0000004e/head//4=0,4194304,a0a2a0bb/100000190e8.00000014/head//4=0,4194304,ab43a0bb/100000283bb.00000001/head//4=0,4194304}, attrs_to_read=)) -4314> 2014-11-08 18:15:26.141392 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12067 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (3481706912 0 0) 0x1ce64ec0 con 0x1c5c2680 -4313> 2014-11-08 18:15:26.141460 7fc5a6cfe700 5 -- op tracker -- seq: 47144, time: 2014-11-08 18:15:26.139022, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) -4312> 2014-11-08 18:15:26.141471 7fc5a6cfe700 5 -- op tracker -- seq: 47144, time: 2014-11-08 18:15:26.139026, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) -4311> 2014-11-08 18:15:26.141476 7fc5a6cfe700 5 -- op tracker -- seq: 47144, time: 2014-11-08 18:15:26.139089, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) -4310> 2014-11-08 18:15:26.141481 7fc5a6cfe700 5 -- op tracker -- seq: 47144, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) -4309> 2014-11-08 18:15:26.141580 7fc5aeb13700 5 -- op tracker -- seq: 47144, time: 2014-11-08 18:15:26.141580, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) -4308> 2014-11-08 18:15:26.192697 7fc5a0f29700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.17:6809/32391 pipe(0x215a5280 sd=22 :6800 s=2 pgs=1431 cs=1 l=1 c=0x259944c0).reader couldn't read tag, (0) Success -4307> 2014-11-08 18:15:26.192757 7fc5a0f29700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.17:6809/32391 pipe(0x215a5280 sd=22 :6800 s=2 pgs=1431 cs=1 l=1 c=0x259944c0).fault (0) Success -4306> 2014-11-08 18:15:26.192953 7fc5bb32c700 1 osd.9 5773 ms_handle_reset con 0x259944c0 session 0x211d4d80 -4305> 2014-11-08 18:15:26.197921 7fc5a0f29700 1 -- 192.168.1.18:6800/4928 >> :/0 pipe(0x2244e880 sd=22 :6800 s=0 pgs=0 cs=0 l=0 c=0x1dbe2c00).accept sd=22 192.168.1.17:39154/0 -4304> 2014-11-08 18:15:26.215436 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25200, attrs_read=0)) v1 -- ?+0 0x21a48500 con 0x1c5c2680 -4303> 2014-11-08 18:15:26.215488 7fc5aeb13700 5 -- op tracker -- seq: 47144, time: 2014-11-08 18:15:26.215487, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25200, to_read={51954227/10000029fee.0000002a/head//4=0,4194304}, attrs_to_read=)) -4302> 2014-11-08 18:15:26.245916 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8596 ==== MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (2488539061 0 0) 0x1d3ac200 con 0x1c5c0580 -4301> 2014-11-08 18:15:26.246196 7fc5a152e700 5 -- op tracker -- seq: 47145, time: 2014-11-08 18:15:26.098406, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4300> 2014-11-08 18:15:26.246216 7fc5a152e700 5 -- op tracker -- seq: 47145, time: 2014-11-08 18:15:26.098413, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4299> 2014-11-08 18:15:26.246230 7fc5a152e700 5 -- op tracker -- seq: 47145, time: 2014-11-08 18:15:26.245905, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4298> 2014-11-08 18:15:26.246243 7fc5a152e700 5 -- op tracker -- seq: 47145, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4297> 2014-11-08 18:15:26.246290 7fc5afb15700 5 -- op tracker -- seq: 47145, time: 2014-11-08 18:15:26.246290, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4296> 2014-11-08 18:15:26.276889 7fc5a152e700 5 -- op tracker -- seq: 47145, time: 2014-11-08 18:15:26.276889, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(490d8b16/1000001a7b0.0000003a/head//4, version: 619'2163, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(490d8b16/1000001a7b0.0000003a/head//4@619'2163, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(7abd8b16/1000002aff9.00000035/head//4, version: 4157'7873, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7abd8b16/1000002aff9.00000035/head//4@4157'7873, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6ffe8b16/1000002b459.0000000c/head//4, version: 3473'6469, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ffe8b16/1000002b459.0000000c/head//4@3473'6469, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4295> 2014-11-08 18:15:26.414150 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 510 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.411754) v2 ==== 47+0+0 (565686879 0 0) 0x1e6ac920 con 0x1eb9d800 -4294> 2014-11-08 18:15:26.414179 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.411754) v2 -- ?+0 0x1caadfa0 con 0x1eb9d800 -4293> 2014-11-08 18:15:26.414204 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 510 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.411754) v2 ==== 47+0+0 (565686879 0 0) 0x1e6ae540 con 0x1eb9c8e0 -4292> 2014-11-08 18:15:26.414213 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.411754) v2 -- ?+0 0x234e6ae0 con 0x1eb9c8e0 -4291> 2014-11-08 18:15:26.427207 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3750 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.426560) v2 ==== 47+0+0 (3327448892 0 0) 0x223494a0 con 0x1cce3180 -4290> 2014-11-08 18:15:26.427262 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.426560) v2 -- ?+0 0x1e6ac920 con 0x1cce3180 -4289> 2014-11-08 18:15:26.427358 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3750 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.426560) v2 ==== 47+0+0 (3327448892 0 0) 0x2127fda0 con 0x1cce3700 -4288> 2014-11-08 18:15:26.427370 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.426560) v2 -- ?+0 0x1e6ae540 con 0x1cce3700 -4287> 2014-11-08 18:15:26.530827 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x24959fe0 con 0x1c5c0580 -4286> 2014-11-08 18:15:26.530891 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495a1c0 con 0x1c5c0580 -4285> 2014-11-08 18:15:26.530898 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495eae0 con 0x1c5c0580 -4284> 2014-11-08 18:15:26.623596 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.13s2 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=be8fdd13/1000002b2f6.00000008/head//4,e89fdd13/1000002b51d.0000000c/head//4,4cdfdd13/1000002982d.00000080/head//4,1d10ed13/10000019b94.00000084/head//4)) v1 -- ?+0 0x2287e780 con 0x1e50aec0 -4283> 2014-11-08 18:15:26.623847 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2287e9c0 con 0x42c86e0 -4282> 2014-11-08 18:15:26.623872 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.13s0( v 5734'8117 (3458'5080,5734'8117] local-les=5760 n=7618 ec=195 les/c 5760/4876 5759/5759/5728) [9,5,2]/[9,2147483647,5] r=0 lpr=5759 pi=4872-5758/40 rops=4 bft=2(2),5(1) crt=5689'8082 lcod 5734'8116 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 3c50ed13/1000002adf9.00000067/head//4 -4281> 2014-11-08 18:15:26.623884 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2287e9c0 con 0x42c86e0 -4280> 2014-11-08 18:15:26.623902 7fc5b9328700 5 -- op tracker -- seq: 47146, time: 2014-11-08 18:15:26.623865, event: header_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) -4279> 2014-11-08 18:15:26.623907 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.13s2 e 5773/5773 lb 7e7edd13/1000002b5ea.000004c4/head//4) v3 -- ?+0 0x213f4c80 con 0x1c5c0580 -4278> 2014-11-08 18:15:26.623915 7fc5b9328700 5 -- op tracker -- seq: 47146, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) -4277> 2014-11-08 18:15:26.623920 7fc5b9328700 5 -- op tracker -- seq: 47146, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) -4276> 2014-11-08 18:15:26.623926 7fc5b9328700 5 -- op tracker -- seq: 47146, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) -4275> 2014-11-08 18:15:26.623984 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.13s1 e 5773/5773 lb 7e7edd13/1000002b5ea.000004c4/head//4) v3 -- ?+0 0x213f0900 con 0x1e50aec0 -4274> 2014-11-08 18:15:26.624043 7fc5ac30e700 5 -- op tracker -- seq: 47146, time: 2014-11-08 18:15:26.624043, event: reached_pg, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) -4273> 2014-11-08 18:15:26.690552 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.18s1 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=596d0518/10000019e0c.0000020a/head//4,5d8d0518/1000002a6bd.0000030e/head//4,50cd0518/10000019d87.00000225/head//4,176e0518/1000002bc6d.0000005b/head//4,f4ce0518/1000002a609.00000028/head//4)) v1 -- ?+0 0x2287ec00 con 0x1c5c0580 -4272> 2014-11-08 18:15:26.690592 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2287ee40 con 0x42c86e0 -4271> 2014-11-08 18:15:26.690651 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2287ee40 con 0x42c86e0 -4270> 2014-11-08 18:15:26.690669 7fc5b9328700 5 -- op tracker -- seq: 47147, time: 2014-11-08 18:15:26.690604, event: header_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) -4269> 2014-11-08 18:15:26.690681 7fc5b9328700 5 -- op tracker -- seq: 47147, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) -4268> 2014-11-08 18:15:26.690701 7fc5b9328700 5 -- op tracker -- seq: 47147, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) -4267> 2014-11-08 18:15:26.690708 7fc5b9328700 5 -- op tracker -- seq: 47147, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) -4266> 2014-11-08 18:15:26.690738 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.18s0( v 5734'8403 (3458'5365,5734'8403] local-les=5760 n=7696 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=5 bft=5(2) crt=5734'8390 lcod 5734'8402 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is cede0518/100000199b6.00000036/head//4 -4265> 2014-11-08 18:15:26.690763 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.18s2 e 5773/5773 lb c36d0518/1000002c371.00000000/head//4) v3 -- ?+0 0x1d652d00 con 0x1e50aec0 -4264> 2014-11-08 18:15:26.690826 7fc5aeb13700 5 -- op tracker -- seq: 47147, time: 2014-11-08 18:15:26.690826, event: reached_pg, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) -4263> 2014-11-08 18:15:26.691002 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.84s2 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=c149e484/1000002a5cd.0000007d/head//4)) v1 -- ?+0 0x2287f2c0 con 0x1c5c0580 -4262> 2014-11-08 18:15:26.691015 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2287f500 con 0x42c86e0 -4261> 2014-11-08 18:15:26.691026 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2287f500 con 0x42c86e0 -4260> 2014-11-08 18:15:26.691033 7fc5b9328700 5 -- op tracker -- seq: 47148, time: 2014-11-08 18:15:26.691020, event: header_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) -4259> 2014-11-08 18:15:26.691038 7fc5b9328700 5 -- op tracker -- seq: 47148, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) -4258> 2014-11-08 18:15:26.691040 7fc5b9328700 5 -- op tracker -- seq: 47148, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) -4257> 2014-11-08 18:15:26.691043 7fc5b9328700 5 -- op tracker -- seq: 47148, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) -4256> 2014-11-08 18:15:26.691063 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.84s0( v 5734'8210 (3458'5187,5734'8210] local-les=5760 n=7768 ec=195 les/c 5760/5758 5759/5759/5728) [9,5,2]/[9,2147483647,2] r=0 lpr=5759 pi=5757-5758/1 rops=1 bft=5(1) crt=5734'8196 lcod 5734'8209 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is c399e484/1000002adfe.00000047/head//4 -4255> 2014-11-08 18:15:26.691081 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.84s1 e 5773/5773 lb b009e484/1000001a70f.0000005f/head//4) v3 -- ?+0 0x1d656780 con 0x1e50aec0 -4254> 2014-11-08 18:15:26.691093 7fc5ad310700 5 -- op tracker -- seq: 47148, time: 2014-11-08 18:15:26.691093, event: reached_pg, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) -4253> 2014-11-08 18:15:26.804795 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) v1 -- ?+0 0x44a6e00 con 0x42c86e0 -4252> 2014-11-08 18:15:26.804844 7fc5ad310700 5 -- op tracker -- seq: 47148, time: 2014-11-08 18:15:26.804843, event: done, op: MOSDECSubOpRead(4.84s0 5773 ECSubRead(tid=4426, to_read={c149e484/1000002a5cd.0000007d/head//4=0,4194304}, attrs_to_read=)) -4251> 2014-11-08 18:15:26.804875 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x44a6e00 con 0x42c86e0 -4250> 2014-11-08 18:15:26.804888 7fc5b9328700 5 -- op tracker -- seq: 47149, time: 2014-11-08 18:15:26.804833, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) -4249> 2014-11-08 18:15:26.804893 7fc5b9328700 5 -- op tracker -- seq: 47149, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) -4248> 2014-11-08 18:15:26.804895 7fc5b9328700 5 -- op tracker -- seq: 47149, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) -4247> 2014-11-08 18:15:26.804898 7fc5b9328700 5 -- op tracker -- seq: 47149, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) -4246> 2014-11-08 18:15:26.804946 7fc5afb15700 5 -- op tracker -- seq: 47149, time: 2014-11-08 18:15:26.804946, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) -4245> 2014-11-08 18:15:26.804989 7fc5afb15700 5 -- op tracker -- seq: 47149, time: 2014-11-08 18:15:26.804989, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=0)) -4244> 2014-11-08 18:15:26.902595 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 325 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.900303) v2 ==== 47+0+0 (937204308 0 0) 0x227c9860 con 0x1eb9d6a0 -4243> 2014-11-08 18:15:26.902648 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.900303) v2 -- ?+0 0x223494a0 con 0x1eb9d6a0 -4242> 2014-11-08 18:15:26.902677 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 325 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:26.900303) v2 ==== 47+0+0 (937204308 0 0) 0x239ee180 con 0x1eb9c780 -4241> 2014-11-08 18:15:26.902693 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:26.900303) v2 -- ?+0 0x2127fda0 con 0x1eb9c780 -4240> 2014-11-08 18:15:26.936406 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) v1 -- ?+0 0x224bc380 con 0x42c86e0 -4239> 2014-11-08 18:15:26.936467 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224bc380 con 0x42c86e0 -4238> 2014-11-08 18:15:26.936484 7fc5b9328700 5 -- op tracker -- seq: 47150, time: 2014-11-08 18:15:26.936454, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) -4237> 2014-11-08 18:15:26.936495 7fc5b9328700 5 -- op tracker -- seq: 47150, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) -4236> 2014-11-08 18:15:26.936497 7fc5b9328700 5 -- op tracker -- seq: 47150, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) -4235> 2014-11-08 18:15:26.936499 7fc5b9328700 5 -- op tracker -- seq: 47150, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) -4234> 2014-11-08 18:15:26.936527 7fc5ac30e700 5 -- op tracker -- seq: 47146, time: 2014-11-08 18:15:26.936527, event: done, op: MOSDECSubOpRead(4.13s0 5773 ECSubRead(tid=4424, to_read={be8fdd13/1000002b2f6.00000008/head//4=0,4194304,e89fdd13/1000002b51d.0000000c/head//4=0,4194304,4cdfdd13/1000002982d.00000080/head//4=0,4194304,1d10ed13/10000019b94.00000084/head//4=0,4194304}, attrs_to_read=)) -4233> 2014-11-08 18:15:26.936557 7fc5ac30e700 5 -- op tracker -- seq: 47150, time: 2014-11-08 18:15:26.936557, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) -4232> 2014-11-08 18:15:26.936609 7fc5ac30e700 5 -- op tracker -- seq: 47150, time: 2014-11-08 18:15:26.936609, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=0)) -4231> 2014-11-08 18:15:26.975603 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8597 ==== MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) v1 ==== 2097696+0+0 (781043074 0 0) 0x21a4c880 con 0x1c5c0580 -4230> 2014-11-08 18:15:26.975625 7fc5a152e700 5 -- op tracker -- seq: 47151, time: 2014-11-08 18:15:26.824435, event: header_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) -4229> 2014-11-08 18:15:26.975631 7fc5a152e700 5 -- op tracker -- seq: 47151, time: 2014-11-08 18:15:26.824438, event: throttled, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) -4228> 2014-11-08 18:15:26.975634 7fc5a152e700 5 -- op tracker -- seq: 47151, time: 2014-11-08 18:15:26.975596, event: all_read, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) -4227> 2014-11-08 18:15:26.975636 7fc5a152e700 5 -- op tracker -- seq: 47151, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) -4226> 2014-11-08 18:15:26.975851 7fc5ad310700 5 -- op tracker -- seq: 47151, time: 2014-11-08 18:15:26.975851, event: reached_pg, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) -4225> 2014-11-08 18:15:26.980303 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.84s1 5773 [PushOp(c149e484/1000002a5cd.0000007d/head//4, version: 3035'5052, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c149e484/1000002a5cd.0000007d/head//4@3035'5052, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x229b8800 con 0x1e50aec0 -4224> 2014-11-08 18:15:26.980396 7fc5ad310700 5 -- op tracker -- seq: 47151, time: 2014-11-08 18:15:26.980396, event: done, op: MOSDECSubOpReadReply(4.84s0 5773 ECSubReadReply(tid=4426, attrs_read=1)) -4223> 2014-11-08 18:15:27.045237 7fc5c362f700 5 osd.9 5773 tick -4222> 2014-11-08 18:15:27.045265 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:07.045263) -4221> 2014-11-08 18:15:27.061029 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist [0,0,0,0,0,0,0,0,0,1]) -4220> 2014-11-08 18:15:27.061063 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x2583cce0 con 0x1cae2940 -4219> 2014-11-08 18:15:27.061087 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x259f1680 con 0x1cae2680 -4218> 2014-11-08 18:15:27.061099 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x25ed32a0 con 0x1c5c02c0 -4217> 2014-11-08 18:15:27.061110 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x2495b2a0 con 0x1cae0160 -4216> 2014-11-08 18:15:27.061120 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1d5e4b00 con 0x1cae0f20 -4215> 2014-11-08 18:15:27.061129 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1e535dc0 con 0x1cae0c60 -4214> 2014-11-08 18:15:27.061139 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x227e23a0 con 0x1cae23c0 -4213> 2014-11-08 18:15:27.061148 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1e6ac1a0 con 0x1cae56a0 -4212> 2014-11-08 18:15:27.061158 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1eac9c20 con 0x1cbf9ce0 -4211> 2014-11-08 18:15:27.061167 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x23684380 con 0x1d603020 -4210> 2014-11-08 18:15:27.061177 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1c837bc0 con 0x1dbd9b80 -4209> 2014-11-08 18:15:27.061185 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x2127e180 con 0x1d235d80 -4208> 2014-11-08 18:15:27.061194 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x22991c20 con 0x1cae1fa0 -4207> 2014-11-08 18:15:27.061203 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1d1b1fe0 con 0x1cae1ce0 -4206> 2014-11-08 18:15:27.061212 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x22859a40 con 0x1cae1340 -4205> 2014-11-08 18:15:27.061221 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1d045fa0 con 0x1cae1080 -4204> 2014-11-08 18:15:27.061230 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x25838780 con 0x1cae1a20 -4203> 2014-11-08 18:15:27.061273 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:27.061060) v2 -- ?+0 0x1d634920 con 0x1cae2c00 -4202> 2014-11-08 18:15:27.061317 7fc5aa30a700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:07.061315) -4201> 2014-11-08 18:15:27.061577 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x2286c380 con 0x1c5c02c0 -4200> 2014-11-08 18:15:27.061744 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x22e9e360 con 0x1cae23c0 -4199> 2014-11-08 18:15:27.061912 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x22996540 con 0x1cae56a0 -4198> 2014-11-08 18:15:27.061926 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x2285f440 con 0x1cae0160 -4197> 2014-11-08 18:15:27.061935 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x23921e00 con 0x1cae0f20 -4196> 2014-11-08 18:15:27.061944 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x23925280 con 0x1cae0c60 -4195> 2014-11-08 18:15:27.061953 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x23686ea0 con 0x1cae2940 -4194> 2014-11-08 18:15:27.061962 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x1d5e6ae0 con 0x1cae2680 -4193> 2014-11-08 18:15:27.062083 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 327 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x1eae5640 con 0x1d235d80 -4192> 2014-11-08 18:15:27.062098 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x1eac94a0 con 0x1cae1ce0 -4191> 2014-11-08 18:15:27.062109 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 327 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x227cc380 con 0x1dbd9b80 -4190> 2014-11-08 18:15:27.062210 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) v1 -- ?+0 0x21838a00 con 0x42c86e0 -4189> 2014-11-08 18:15:27.062248 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(19530db5/1000002002e.00000000/head//4),PushReplyOp(5e930db5/1000002a777.000000a2/head//4),PushReplyOp(11040db5/1000001b4ed.00000608/head//4),PushReplyOp(7b240db5/10000019b1e.0000038f/head//4),PushReplyOp(45340db5/10000019b35.000007b5/head//4)]) v2 -- ?+0 0x1d3ad000 con 0x1c5c0580 -4188> 2014-11-08 18:15:27.062301 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(a99f90b7/1000001cfbf.00000000/head//4)]) v2 -- ?+0 0x24847200 con 0x1e50aec0 -4187> 2014-11-08 18:15:27.062321 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(490d8b16/1000001a7b0.0000003a/head//4),PushReplyOp(7abd8b16/1000002aff9.00000035/head//4),PushReplyOp(6ffe8b16/1000002b459.0000000c/head//4)]) v2 -- ?+0 0x1d3ab000 con 0x1c5c0580 -4186> 2014-11-08 18:15:27.062425 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x21838a00 con 0x42c86e0 -4185> 2014-11-08 18:15:27.062436 7fc5b9328700 5 -- op tracker -- seq: 47152, time: 2014-11-08 18:15:27.062222, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) -4184> 2014-11-08 18:15:27.062440 7fc5b9328700 5 -- op tracker -- seq: 47152, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) -4183> 2014-11-08 18:15:27.062442 7fc5b9328700 5 -- op tracker -- seq: 47152, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) -4182> 2014-11-08 18:15:27.062444 7fc5b9328700 5 -- op tracker -- seq: 47152, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) -4181> 2014-11-08 18:15:27.062530 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x1eac85a0 con 0x1cae1fa0 -4180> 2014-11-08 18:15:27.062544 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x2257d0a0 con 0x1cae1a20 -4179> 2014-11-08 18:15:27.062554 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x2127b660 con 0x1cae2c00 -4178> 2014-11-08 18:15:27.062576 7fc5ac30e700 5 -- op tracker -- seq: 47152, time: 2014-11-08 18:15:27.062576, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) -4177> 2014-11-08 18:15:27.062677 7fc5ac30e700 5 -- op tracker -- seq: 47152, time: 2014-11-08 18:15:27.062677, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=0)) -4176> 2014-11-08 18:15:27.063844 7fc5aeb13700 5 -- op tracker -- seq: 47147, time: 2014-11-08 18:15:27.063844, event: done, op: MOSDECSubOpRead(4.18s0 5773 ECSubRead(tid=4425, to_read={596d0518/10000019e0c.0000020a/head//4=0,4194304,5d8d0518/1000002a6bd.0000030e/head//4=0,4194304,50cd0518/10000019d87.00000225/head//4=0,4194304,176e0518/1000002bc6d.0000005b/head//4=0,4194304,f4ce0518/1000002a609.00000028/head//4=0,4194304}, attrs_to_read=)) -4175> 2014-11-08 18:15:27.064449 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8598 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) v3 ==== 757+0+0 (1504833974 0 0) 0x213f4c80 con 0x1c5c0580 -4174> 2014-11-08 18:15:27.064469 7fc5a152e700 5 -- op tracker -- seq: 47153, time: 2014-11-08 18:15:27.064382, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4173> 2014-11-08 18:15:27.064480 7fc5a152e700 5 -- op tracker -- seq: 47153, time: 2014-11-08 18:15:27.064384, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4172> 2014-11-08 18:15:27.064489 7fc5a152e700 5 -- op tracker -- seq: 47153, time: 2014-11-08 18:15:27.064434, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4171> 2014-11-08 18:15:27.064498 7fc5a152e700 5 -- op tracker -- seq: 47153, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4170> 2014-11-08 18:15:27.064532 7fc5adb11700 5 -- op tracker -- seq: 47153, time: 2014-11-08 18:15:27.064532, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4169> 2014-11-08 18:15:27.064541 7fc5adb11700 5 -- op tracker -- seq: 47153, time: 2014-11-08 18:15:27.064541, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4168> 2014-11-08 18:15:27.064667 7fc5a152e700 5 -- op tracker -- seq: 47153, time: 2014-11-08 18:15:27.064667, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 45340db5/10000019b35.000007b5/head//4) -4167> 2014-11-08 18:15:27.066854 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8599 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) v3 ==== 757+0+0 (3747126003 0 0) 0x213f4c80 con 0x1c5c0580 -4166> 2014-11-08 18:15:27.066895 7fc5a152e700 5 -- op tracker -- seq: 47154, time: 2014-11-08 18:15:27.066721, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4165> 2014-11-08 18:15:27.066910 7fc5a152e700 5 -- op tracker -- seq: 47154, time: 2014-11-08 18:15:27.066725, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4164> 2014-11-08 18:15:27.066919 7fc5a152e700 5 -- op tracker -- seq: 47154, time: 2014-11-08 18:15:27.066829, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4163> 2014-11-08 18:15:27.066929 7fc5a152e700 5 -- op tracker -- seq: 47154, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4162> 2014-11-08 18:15:27.066974 7fc5afb15700 5 -- op tracker -- seq: 47154, time: 2014-11-08 18:15:27.066974, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4161> 2014-11-08 18:15:27.066984 7fc5afb15700 5 -- op tracker -- seq: 47154, time: 2014-11-08 18:15:27.066984, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4160> 2014-11-08 18:15:27.067163 7fc5a152e700 5 -- op tracker -- seq: 47154, time: 2014-11-08 18:15:27.067163, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 6ffe8b16/1000002b459.0000000c/head//4) -4159> 2014-11-08 18:15:27.069889 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x1d040780 con 0x1cae1080 -4158> 2014-11-08 18:15:27.069927 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3740 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:27.061060) v2 ==== 47+0+0 (750580796 0 0) 0x2583a580 con 0x1cae1340 -4157> 2014-11-08 18:15:27.089303 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2283 ==== pg_trim(4.84 to 5734'8210 e5773) v1 ==== 34+0+0 (869693373 0 0) 0x249590e0 con 0x1e50aec0 -4156> 2014-11-08 18:15:27.089341 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089198, event: header_read, op: pg_trim(4.84 to 5734'8210 e5773) -4155> 2014-11-08 18:15:27.089350 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089201, event: throttled, op: pg_trim(4.84 to 5734'8210 e5773) -4154> 2014-11-08 18:15:27.089353 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089278, event: all_read, op: pg_trim(4.84 to 5734'8210 e5773) -4153> 2014-11-08 18:15:27.089356 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089336, event: dispatched, op: pg_trim(4.84 to 5734'8210 e5773) -4152> 2014-11-08 18:15:27.089359 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089359, event: waiting_for_osdmap, op: pg_trim(4.84 to 5734'8210 e5773) -4151> 2014-11-08 18:15:27.089368 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089368, event: started, op: pg_trim(4.84 to 5734'8210 e5773) -4150> 2014-11-08 18:15:27.089374 7fc5b9b29700 5 -- op tracker -- seq: 47155, time: 2014-11-08 18:15:27.089373, event: done, op: pg_trim(4.84 to 5734'8210 e5773) -4149> 2014-11-08 18:15:27.093783 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12068 ==== MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (142007203 0 0) 0x1ce65e80 con 0x1c5c2680 -4148> 2014-11-08 18:15:27.093815 7fc5a6cfe700 5 -- op tracker -- seq: 47156, time: 2014-11-08 18:15:27.093695, event: header_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) -4147> 2014-11-08 18:15:27.093823 7fc5a6cfe700 5 -- op tracker -- seq: 47156, time: 2014-11-08 18:15:27.093700, event: throttled, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) -4146> 2014-11-08 18:15:27.093826 7fc5a6cfe700 5 -- op tracker -- seq: 47156, time: 2014-11-08 18:15:27.093776, event: all_read, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) -4145> 2014-11-08 18:15:27.093830 7fc5a6cfe700 5 -- op tracker -- seq: 47156, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) -4144> 2014-11-08 18:15:27.093885 7fc5ac30e700 5 -- op tracker -- seq: 47156, time: 2014-11-08 18:15:27.093885, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) -4143> 2014-11-08 18:15:27.172880 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.27s0 5773 ECSubReadReply(tid=25201, attrs_read=0)) v1 -- ?+0 0x21838a00 con 0x1c5c2680 -4142> 2014-11-08 18:15:27.175814 7fc5ac30e700 5 -- op tracker -- seq: 47156, time: 2014-11-08 18:15:27.175813, event: done, op: MOSDECSubOpRead(4.27s1 5773 ECSubRead(tid=25201, to_read={c0a74227/1000002ada3.00000095/head//4=0,4194304}, attrs_to_read=)) -4141> 2014-11-08 18:15:27.197459 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2284 ==== pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) v3 ==== 757+0+0 (2432462773 0 0) 0x23ced100 con 0x1e50aec0 -4140> 2014-11-08 18:15:27.197489 7fc5a56ef700 5 -- op tracker -- seq: 47157, time: 2014-11-08 18:15:27.197370, event: header_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4139> 2014-11-08 18:15:27.197499 7fc5a56ef700 5 -- op tracker -- seq: 47157, time: 2014-11-08 18:15:27.197373, event: throttled, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4138> 2014-11-08 18:15:27.197513 7fc5a56ef700 5 -- op tracker -- seq: 47157, time: 2014-11-08 18:15:27.197451, event: all_read, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4137> 2014-11-08 18:15:27.197517 7fc5a56ef700 5 -- op tracker -- seq: 47157, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4136> 2014-11-08 18:15:27.197557 7fc5af314700 5 -- op tracker -- seq: 47157, time: 2014-11-08 18:15:27.197557, event: reached_pg, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4135> 2014-11-08 18:15:27.197603 7fc5af314700 5 -- op tracker -- seq: 47157, time: 2014-11-08 18:15:27.197602, event: started, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4134> 2014-11-08 18:15:27.197653 7fc5af314700 5 -- op tracker -- seq: 47157, time: 2014-11-08 18:15:27.197653, event: done, op: pg_backfill(progress 4.b7s1 e 5773/5773 lb a99f90b7/1000001cfbf.00000000/head//4) -4133> 2014-11-08 18:15:27.220392 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1506 ==== pg_stats_ack(3 pgs tid 1331) v1 ==== 91+0+0 (1063841397 0 0) 0x2371b0c0 con 0x44ed6a0 -4132> 2014-11-08 18:15:27.414409 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8600 ==== MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) v1 ==== 10488220+0+0 (4230432045 0 0) 0x21a49180 con 0x1c5c0580 -4131> 2014-11-08 18:15:27.414448 7fc5a152e700 5 -- op tracker -- seq: 47158, time: 2014-11-08 18:15:27.074588, event: header_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) -4130> 2014-11-08 18:15:27.414462 7fc5a152e700 5 -- op tracker -- seq: 47158, time: 2014-11-08 18:15:27.074592, event: throttled, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) -4129> 2014-11-08 18:15:27.414471 7fc5a152e700 5 -- op tracker -- seq: 47158, time: 2014-11-08 18:15:27.414383, event: all_read, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) -4128> 2014-11-08 18:15:27.414480 7fc5a152e700 5 -- op tracker -- seq: 47158, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) -4127> 2014-11-08 18:15:27.414521 7fc5aeb13700 5 -- op tracker -- seq: 47158, time: 2014-11-08 18:15:27.414521, event: reached_pg, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) -4126> 2014-11-08 18:15:27.460610 7fc5aeb13700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.18s2 5773 [PushOp(596d0518/10000019e0c.0000020a/head//4, version: 397'1586, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(596d0518/10000019e0c.0000020a/head//4@397'1586, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5d8d0518/1000002a6bd.0000030e/head//4, version: 3458'5315, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5d8d0518/1000002a6bd.0000030e/head//4@3458'5315, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(50cd0518/10000019d87.00000225/head//4, version: 397'1537, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(50cd0518/10000019d87.00000225/head//4@397'1537, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(176e0518/1000002bc6d.0000005b/head//4, version: 3478'7584, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(176e0518/1000002bc6d.0000005b/head/ /4@3478'7584, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f4ce0518/1000002a609.00000028/head//4, version: 3159'5224, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f4ce0518/1000002a609.00000028/head//4@3159'5224, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x2393c400 con 0x1e50aec0 -4125> 2014-11-08 18:15:27.480501 7fc5aeb13700 5 -- op tracker -- seq: 47158, time: 2014-11-08 18:15:27.480500, event: done, op: MOSDECSubOpReadReply(4.18s0 5773 ECSubReadReply(tid=4425, attrs_read=5)) -4124> 2014-11-08 18:15:27.495292 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8601 ==== MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (3544054413 0 0) 0x1d3ab000 con 0x1c5c0580 -4123> 2014-11-08 18:15:27.495335 7fc5a152e700 5 -- op tracker -- seq: 47159, time: 2014-11-08 18:15:27.416728, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4122> 2014-11-08 18:15:27.495347 7fc5a152e700 5 -- op tracker -- seq: 47159, time: 2014-11-08 18:15:27.416730, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4121> 2014-11-08 18:15:27.495353 7fc5a152e700 5 -- op tracker -- seq: 47159, time: 2014-11-08 18:15:27.495256, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4120> 2014-11-08 18:15:27.495359 7fc5a152e700 5 -- op tracker -- seq: 47159, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4119> 2014-11-08 18:15:27.495560 7fc5b0316700 5 -- op tracker -- seq: 47159, time: 2014-11-08 18:15:27.495560, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4118> 2014-11-08 18:15:27.502327 7fc5a152e700 5 -- op tracker -- seq: 47159, time: 2014-11-08 18:15:27.502327, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(68540db5/1000002a749.0000008b/head//4, version: 3458'5316, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(68540db5/1000002a749.0000008b/head//4@3458'5316, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -4117> 2014-11-08 18:15:27.836043 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2285 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22d51a40 con 0x1e50aec0 -4116> 2014-11-08 18:15:27.836083 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.835949, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4115> 2014-11-08 18:15:27.836092 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.835952, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4114> 2014-11-08 18:15:27.836097 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.836018, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4113> 2014-11-08 18:15:27.836101 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.836076, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4112> 2014-11-08 18:15:27.836107 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.836107, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4111> 2014-11-08 18:15:27.836121 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.836121, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4110> 2014-11-08 18:15:27.836129 7fc5b9b29700 5 -- op tracker -- seq: 47160, time: 2014-11-08 18:15:27.836129, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4109> 2014-11-08 18:15:27.836141 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2286 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22d53660 con 0x1e50aec0 -4108> 2014-11-08 18:15:27.836150 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836042, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4107> 2014-11-08 18:15:27.836154 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836042, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4106> 2014-11-08 18:15:27.836159 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836061, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4105> 2014-11-08 18:15:27.836163 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836149, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4104> 2014-11-08 18:15:27.836167 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836167, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4103> 2014-11-08 18:15:27.836173 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836173, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4102> 2014-11-08 18:15:27.836178 7fc5b9b29700 5 -- op tracker -- seq: 47161, time: 2014-11-08 18:15:27.836178, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4101> 2014-11-08 18:15:27.836185 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2287 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x22d56ea0 con 0x1e50aec0 -4100> 2014-11-08 18:15:27.836194 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836090, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4099> 2014-11-08 18:15:27.836197 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836091, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4098> 2014-11-08 18:15:27.836199 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836125, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4097> 2014-11-08 18:15:27.836202 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836193, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4096> 2014-11-08 18:15:27.836204 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836204, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4095> 2014-11-08 18:15:27.836208 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836208, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4094> 2014-11-08 18:15:27.836211 7fc5b9b29700 5 -- op tracker -- seq: 47162, time: 2014-11-08 18:15:27.836211, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4093> 2014-11-08 18:15:27.836217 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2288 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x23c30960 con 0x1e50aec0 -4092> 2014-11-08 18:15:27.836225 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836137, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4091> 2014-11-08 18:15:27.836230 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836138, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4090> 2014-11-08 18:15:27.836234 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836171, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4089> 2014-11-08 18:15:27.836238 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836224, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4088> 2014-11-08 18:15:27.836243 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836243, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4087> 2014-11-08 18:15:27.836248 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836248, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4086> 2014-11-08 18:15:27.836253 7fc5b9b29700 5 -- op tracker -- seq: 47163, time: 2014-11-08 18:15:27.836253, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4085> 2014-11-08 18:15:27.836272 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2289 ==== pg_trim(4.18 to 5734'8403 e5773) v1 ==== 34+0+0 (40556134 0 0) 0x23c37800 con 0x1e50aec0 -4084> 2014-11-08 18:15:27.836283 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836181, event: header_read, op: pg_trim(4.18 to 5734'8403 e5773) -4083> 2014-11-08 18:15:27.836286 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836182, event: throttled, op: pg_trim(4.18 to 5734'8403 e5773) -4082> 2014-11-08 18:15:27.836288 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836264, event: all_read, op: pg_trim(4.18 to 5734'8403 e5773) -4081> 2014-11-08 18:15:27.836291 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836282, event: dispatched, op: pg_trim(4.18 to 5734'8403 e5773) -4080> 2014-11-08 18:15:27.836293 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836293, event: waiting_for_osdmap, op: pg_trim(4.18 to 5734'8403 e5773) -4079> 2014-11-08 18:15:27.836297 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836297, event: started, op: pg_trim(4.18 to 5734'8403 e5773) -4078> 2014-11-08 18:15:27.836300 7fc5b9b29700 5 -- op tracker -- seq: 47164, time: 2014-11-08 18:15:27.836300, event: done, op: pg_trim(4.18 to 5734'8403 e5773) -4077> 2014-11-08 18:15:27.925800 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2495bde0 con 0x1c5c0580 -4076> 2014-11-08 18:15:27.925913 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(68540db5/1000002a749.0000008b/head//4)]) v2 -- ?+0 0x1c6a7c00 con 0x1c5c0580 -4075> 2014-11-08 18:15:27.961670 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8602 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) v3 ==== 757+0+0 (1683410647 0 0) 0x213f4c80 con 0x1c5c0580 -4074> 2014-11-08 18:15:27.961718 7fc5a152e700 5 -- op tracker -- seq: 47165, time: 2014-11-08 18:15:27.961578, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4073> 2014-11-08 18:15:27.961729 7fc5a152e700 5 -- op tracker -- seq: 47165, time: 2014-11-08 18:15:27.961582, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4072> 2014-11-08 18:15:27.961734 7fc5a152e700 5 -- op tracker -- seq: 47165, time: 2014-11-08 18:15:27.961661, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4071> 2014-11-08 18:15:27.961738 7fc5a152e700 5 -- op tracker -- seq: 47165, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4070> 2014-11-08 18:15:27.961800 7fc5adb11700 5 -- op tracker -- seq: 47165, time: 2014-11-08 18:15:27.961800, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4069> 2014-11-08 18:15:27.961815 7fc5adb11700 5 -- op tracker -- seq: 47165, time: 2014-11-08 18:15:27.961815, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4068> 2014-11-08 18:15:27.961866 7fc5adb11700 5 -- op tracker -- seq: 47165, time: 2014-11-08 18:15:27.961866, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 68540db5/1000002a749.0000008b/head//4) -4067> 2014-11-08 18:15:27.977602 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2290 ==== MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) v1 ==== 8390589+0+0 (2575814930 0 0) 0x224bdc80 con 0x1e50aec0 -4066> 2014-11-08 18:15:27.977661 7fc5a56ef700 5 -- op tracker -- seq: 47166, time: 2014-11-08 18:15:27.882519, event: header_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) -4065> 2014-11-08 18:15:27.977670 7fc5a56ef700 5 -- op tracker -- seq: 47166, time: 2014-11-08 18:15:27.882521, event: throttled, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) -4064> 2014-11-08 18:15:27.977675 7fc5a56ef700 5 -- op tracker -- seq: 47166, time: 2014-11-08 18:15:27.977590, event: all_read, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) -4063> 2014-11-08 18:15:27.977678 7fc5a56ef700 5 -- op tracker -- seq: 47166, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) -4062> 2014-11-08 18:15:27.977721 7fc5ac30e700 5 -- op tracker -- seq: 47166, time: 2014-11-08 18:15:27.977721, event: reached_pg, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) -4061> 2014-11-08 18:15:28.006153 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.13s2 5773 [PushOp(be8fdd13/1000002b2f6.00000008/head//4, version: 3468'6086, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be8fdd13/1000002b2f6.00000008/head//4@3468'6086, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e89fdd13/1000002b51d.0000000c/head//4, version: 3473'6290, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e89fdd13/1000002b51d.0000000c/head//4@3473'6290, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4cdfdd13/1000002982d.00000080/head//4, version: 3033'4303, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdfdd13/1000002982d.00000080/head//4@3033'4303, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1d10ed13/10000019b94.00000084/head//4, version: 397'1188, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1d10ed13/10000019b94.00000084/ head//4@397'1188, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x256ffc00 con 0x1c5c0580 -4060> 2014-11-08 18:15:28.006216 7fc5ac30e700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.13s1 5773 [PushOp(be8fdd13/1000002b2f6.00000008/head//4, version: 3468'6086, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be8fdd13/1000002b2f6.00000008/head//4@3468'6086, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e89fdd13/1000002b51d.0000000c/head//4, version: 3473'6290, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e89fdd13/1000002b51d.0000000c/head//4@3473'6290, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(4cdfdd13/1000002982d.00000080/head//4, version: 3033'4303, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4cdfdd13/1000002982d.00000080/head//4@3033'4303, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(1d10ed13/10000019b94.00000084/head//4, version: 397'1188, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1d10ed13/ 10000019b94.00000084/head//4@397'1188, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x256fdc00 con 0x1e50aec0 -4059> 2014-11-08 18:15:28.014932 7fc5ac30e700 5 -- op tracker -- seq: 47166, time: 2014-11-08 18:15:28.014931, event: done, op: MOSDECSubOpReadReply(4.13s0 5773 ECSubReadReply(tid=4424, attrs_read=4)) -4058> 2014-11-08 18:15:28.049464 7fc5c362f700 5 osd.9 5773 tick -4057> 2014-11-08 18:15:28.049496 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:08.049493) -4056> 2014-11-08 18:15:28.115207 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 511 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:28.112578) v2 ==== 47+0+0 (738495493 0 0) 0x1e6ae360 con 0x1eb9c8e0 -4055> 2014-11-08 18:15:28.115234 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.112578) v2 -- ?+0 0x239ee180 con 0x1eb9c8e0 -4054> 2014-11-08 18:15:28.115256 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 511 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:28.112578) v2 ==== 47+0+0 (738495493 0 0) 0x1e6ac740 con 0x1eb9d800 -4053> 2014-11-08 18:15:28.115266 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.112578) v2 -- ?+0 0x227c9860 con 0x1eb9d800 -4052> 2014-11-08 18:15:28.161719 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -4051> 2014-11-08 18:15:28.161747 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x2495f260 con 0x1cae2940 -4050> 2014-11-08 18:15:28.161769 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x25ed1860 con 0x1cae2680 -4049> 2014-11-08 18:15:28.161778 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x22859c20 con 0x1c5c02c0 -4048> 2014-11-08 18:15:28.161787 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1c8341a0 con 0x1cae0160 -4047> 2014-11-08 18:15:28.161796 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1d5e4740 con 0x1cae0f20 -4046> 2014-11-08 18:15:28.161806 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x23682580 con 0x1cae0c60 -4045> 2014-11-08 18:15:28.161814 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1d1b30c0 con 0x1cae23c0 -4044> 2014-11-08 18:15:28.161824 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1eacd820 con 0x1cae56a0 -4043> 2014-11-08 18:15:28.163437 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1d040000 con 0x1cbf9ce0 -4042> 2014-11-08 18:15:28.163448 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x229912c0 con 0x1d603020 -4041> 2014-11-08 18:15:28.163457 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x2127d460 con 0x1dbd9b80 -4040> 2014-11-08 18:15:28.163466 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1e6ac380 con 0x1d235d80 -4039> 2014-11-08 18:15:28.163475 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x227e7bc0 con 0x1cae1fa0 -4038> 2014-11-08 18:15:28.163484 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1e5323a0 con 0x1cae1ce0 -4037> 2014-11-08 18:15:28.163493 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1d042580 con 0x1cae1340 -4036> 2014-11-08 18:15:28.163502 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x259f0000 con 0x1cae1080 -4035> 2014-11-08 18:15:28.163510 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x2127bde0 con 0x1cae1a20 -4034> 2014-11-08 18:15:28.163519 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:28.161744) v2 -- ?+0 0x1d1b5fa0 con 0x1cae2c00 -4033> 2014-11-08 18:15:28.163529 7fc5aa30a700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:08.163527) -4032> 2014-11-08 18:15:28.164233 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x1d5e0f00 con 0x1cae2680 -4031> 2014-11-08 18:15:28.164278 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x22990f00 con 0x1cae56a0 -4030> 2014-11-08 18:15:28.164289 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x2286eea0 con 0x1c5c02c0 -4029> 2014-11-08 18:15:28.164298 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x2285ddc0 con 0x1cae0160 -4028> 2014-11-08 18:15:28.164307 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x4530780 con 0x1cae0c60 -4027> 2014-11-08 18:15:28.164315 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x1eaceea0 con 0x1cae1fa0 -4026> 2014-11-08 18:15:28.164325 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x23685280 con 0x1cae2940 -4025> 2014-11-08 18:15:28.164382 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x213bd280 con 0x1cae0f20 -4024> 2014-11-08 18:15:28.164395 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x1eacb0c0 con 0x1cae1ce0 -4023> 2014-11-08 18:15:28.164825 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 328 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x1eae1a40 con 0x1d235d80 -4022> 2014-11-08 18:15:28.164841 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x2583f260 con 0x1cae1340 -4021> 2014-11-08 18:15:28.164851 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x1d0414a0 con 0x1cae1080 -4020> 2014-11-08 18:15:28.164859 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x2257b0c0 con 0x1cae1a20 -4019> 2014-11-08 18:15:28.164868 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x2127cec0 con 0x1cae2c00 -4018> 2014-11-08 18:15:28.164877 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3741 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x22e9c740 con 0x1cae23c0 -4017> 2014-11-08 18:15:28.164885 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 328 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.161744) v2 ==== 47+0+0 (3090390036 0 0) 0x227c85a0 con 0x1dbd9b80 -4016> 2014-11-08 18:15:28.178303 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2291 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x23c34740 con 0x1e50aec0 -4015> 2014-11-08 18:15:28.178339 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178154, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -4014> 2014-11-08 18:15:28.178347 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178157, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -4013> 2014-11-08 18:15:28.178350 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178280, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -4012> 2014-11-08 18:15:28.178352 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178333, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -4011> 2014-11-08 18:15:28.178355 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178355, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -4010> 2014-11-08 18:15:28.178363 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178363, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -4009> 2014-11-08 18:15:28.178368 7fc5b9b29700 5 -- op tracker -- seq: 47167, time: 2014-11-08 18:15:28.178368, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -4008> 2014-11-08 18:15:28.178476 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2292 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x23c303c0 con 0x1e50aec0 -4007> 2014-11-08 18:15:28.178485 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178385, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -4006> 2014-11-08 18:15:28.178488 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178385, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -4005> 2014-11-08 18:15:28.178491 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178407, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -4004> 2014-11-08 18:15:28.178493 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178483, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -4003> 2014-11-08 18:15:28.178496 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178496, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -4002> 2014-11-08 18:15:28.178499 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178499, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -4001> 2014-11-08 18:15:28.178503 7fc5b9b29700 5 -- op tracker -- seq: 47168, time: 2014-11-08 18:15:28.178503, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -4000> 2014-11-08 18:15:28.178543 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2293 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x22a23a20 con 0x1e50aec0 -3999> 2014-11-08 18:15:28.178550 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178514, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -3998> 2014-11-08 18:15:28.178554 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178514, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -3997> 2014-11-08 18:15:28.178556 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178535, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -3996> 2014-11-08 18:15:28.178559 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178549, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -3995> 2014-11-08 18:15:28.178561 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178561, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -3994> 2014-11-08 18:15:28.178565 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178565, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -3993> 2014-11-08 18:15:28.178568 7fc5b9b29700 5 -- op tracker -- seq: 47169, time: 2014-11-08 18:15:28.178568, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -3992> 2014-11-08 18:15:28.178618 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2294 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x227cf620 con 0x1e50aec0 -3991> 2014-11-08 18:15:28.178626 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178578, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -3990> 2014-11-08 18:15:28.178629 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178579, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -3989> 2014-11-08 18:15:28.178632 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178611, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -3988> 2014-11-08 18:15:28.178634 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178625, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -3987> 2014-11-08 18:15:28.178637 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178637, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -3986> 2014-11-08 18:15:28.178641 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178641, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -3985> 2014-11-08 18:15:28.178644 7fc5b9b29700 5 -- op tracker -- seq: 47170, time: 2014-11-08 18:15:28.178644, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -3984> 2014-11-08 18:15:28.296151 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8603 ==== MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/ 10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10175018+0+0 (3595190025 0 0) 0x1c6a7c00 con 0x1c5c0580 -3983> 2014-11-08 18:15:28.297202 7fc5a152e700 5 -- op tracker -- seq: 47171, time: 2014-11-08 18:15:28.147277, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3982> 2014-11-08 18:15:28.297235 7fc5a152e700 5 -- op tracker -- seq: 47171, time: 2014-11-08 18:15:28.147283, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3981> 2014-11-08 18:15:28.297260 7fc5a152e700 5 -- op tracker -- seq: 47171, time: 2014-11-08 18:15:28.296021, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3980> 2014-11-08 18:15:28.297308 7fc5a152e700 5 -- op tracker -- seq: 47171, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/ 10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3979> 2014-11-08 18:15:28.297366 7fc5ad310700 5 -- op tracker -- seq: 47171, time: 2014-11-08 18:15:28.297366, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3978> 2014-11-08 18:15:28.297610 7fc5ad310700 5 -- op tracker -- seq: 47171, time: 2014-11-08 18:15:28.297610, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(5f2f8b16/1000002b1f1.0000003c/head//4, version: 4330'8115, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5f2f8b16/1000002b1f1.0000003c/head//4@4330'8115, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(359f8b16/1000002bdb6.0000007f/head//4, version: 4864'8341, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(359f8b16/1000002bdb6.0000007f/head//4@4864'8341, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9b809b16/1000002a4af.00000000/head//4, version: 3458'5443, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(9b809b16/1000002a4af.00000000/head//4@3458'5443, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(e5d09b16/10000028aae.0000000e/head//4, version: 3033'3990, data_included: [0~1781760], data_size: 1781760, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e5d09b16/ 10000028aae.0000000e/head//4@3033'3990, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3563520, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(3319b16/10000019a01.00000033/head//4, version: 397'900, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3319b16/10000019a01.00000033/head//4@397'900, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3977> 2014-11-08 18:15:28.333240 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8604 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x2495bde0 con 0x1c5c0580 -3976> 2014-11-08 18:15:28.333278 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.330327, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -3975> 2014-11-08 18:15:28.333287 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.330331, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -3974> 2014-11-08 18:15:28.333289 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.330403, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -3973> 2014-11-08 18:15:28.333292 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.333272, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -3972> 2014-11-08 18:15:28.333295 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.333295, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -3971> 2014-11-08 18:15:28.333360 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.333360, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -3970> 2014-11-08 18:15:28.333366 7fc5b9b29700 5 -- op tracker -- seq: 47172, time: 2014-11-08 18:15:28.333366, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -3969> 2014-11-08 18:15:28.335716 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8608 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) v2 ==== 305+0+0 (3004400999 0 0) 0x256ffc00 con 0x1c5c0580 -3968> 2014-11-08 18:15:28.335735 7fc5a152e700 5 -- op tracker -- seq: 47173, time: 2014-11-08 18:15:28.335675, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -3967> 2014-11-08 18:15:28.335742 7fc5a152e700 5 -- op tracker -- seq: 47173, time: 2014-11-08 18:15:28.335675, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -3966> 2014-11-08 18:15:28.335746 7fc5a152e700 5 -- op tracker -- seq: 47173, time: 2014-11-08 18:15:28.335707, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -3965> 2014-11-08 18:15:28.335750 7fc5a152e700 5 -- op tracker -- seq: 47173, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -3964> 2014-11-08 18:15:28.335789 7fc5aeb13700 5 -- op tracker -- seq: 47173, time: 2014-11-08 18:15:28.335789, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -3963> 2014-11-08 18:15:28.335831 7fc5a152e700 5 -- op tracker -- seq: 47173, time: 2014-11-08 18:15:28.335831, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -3962> 2014-11-08 18:15:28.340845 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8605 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x2495eae0 con 0x1c5c0580 -3961> 2014-11-08 18:15:28.340872 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.332292, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -3960> 2014-11-08 18:15:28.340878 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.332295, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -3959> 2014-11-08 18:15:28.340881 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.335521, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -3958> 2014-11-08 18:15:28.340883 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.340867, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -3957> 2014-11-08 18:15:28.340887 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.340887, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -3956> 2014-11-08 18:15:28.340893 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.340893, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -3955> 2014-11-08 18:15:28.340898 7fc5b9b29700 5 -- op tracker -- seq: 47174, time: 2014-11-08 18:15:28.340898, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -3954> 2014-11-08 18:15:28.340911 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8606 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x2495a1c0 con 0x1c5c0580 -3953> 2014-11-08 18:15:28.340917 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.335564, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -3952> 2014-11-08 18:15:28.340920 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.335572, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -3951> 2014-11-08 18:15:28.340922 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.335630, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -3950> 2014-11-08 18:15:28.340924 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.340916, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -3949> 2014-11-08 18:15:28.340927 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.340927, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -3948> 2014-11-08 18:15:28.340930 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.340930, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -3947> 2014-11-08 18:15:28.340933 7fc5b9b29700 5 -- op tracker -- seq: 47175, time: 2014-11-08 18:15:28.340933, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -3946> 2014-11-08 18:15:28.340939 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8607 ==== pg_trim(4.13 to 5734'8117 e5773) v1 ==== 34+0+0 (3874812473 0 0) 0x24959fe0 con 0x1c5c0580 -3945> 2014-11-08 18:15:28.340945 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.335643, event: header_read, op: pg_trim(4.13 to 5734'8117 e5773) -3944> 2014-11-08 18:15:28.340947 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.335644, event: throttled, op: pg_trim(4.13 to 5734'8117 e5773) -3943> 2014-11-08 18:15:28.340950 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.335662, event: all_read, op: pg_trim(4.13 to 5734'8117 e5773) -3942> 2014-11-08 18:15:28.340952 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.340944, event: dispatched, op: pg_trim(4.13 to 5734'8117 e5773) -3941> 2014-11-08 18:15:28.340955 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.340954, event: waiting_for_osdmap, op: pg_trim(4.13 to 5734'8117 e5773) -3940> 2014-11-08 18:15:28.340958 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.340958, event: started, op: pg_trim(4.13 to 5734'8117 e5773) -3939> 2014-11-08 18:15:28.340961 7fc5b9b29700 5 -- op tracker -- seq: 47176, time: 2014-11-08 18:15:28.340961, event: done, op: pg_trim(4.13 to 5734'8117 e5773) -3938> 2014-11-08 18:15:28.573199 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8609 ==== MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (1687390154 0 0) 0x256ffc00 con 0x1c5c0580 -3937> 2014-11-08 18:15:28.573259 7fc5a152e700 5 -- op tracker -- seq: 47177, time: 2014-11-08 18:15:28.490398, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3936> 2014-11-08 18:15:28.573283 7fc5a152e700 5 -- op tracker -- seq: 47177, time: 2014-11-08 18:15:28.490401, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3935> 2014-11-08 18:15:28.573302 7fc5a152e700 5 -- op tracker -- seq: 47177, time: 2014-11-08 18:15:28.573187, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3934> 2014-11-08 18:15:28.573321 7fc5a152e700 5 -- op tracker -- seq: 47177, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3933> 2014-11-08 18:15:28.573376 7fc5b0316700 5 -- op tracker -- seq: 47177, time: 2014-11-08 18:15:28.573376, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3932> 2014-11-08 18:15:28.573502 7fc5b0316700 5 -- op tracker -- seq: 47177, time: 2014-11-08 18:15:28.573502, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(2b550db5/1000002b4fc.00000040/head//4, version: 3473'6449, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2b550db5/1000002b4fc.00000040/head//4@3473'6449, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ccd50db5/1000002a4e5.00000005/head//4, version: 3033'4952, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ccd50db5/1000002a4e5.00000005/head//4@3033'4952, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d6f50db5/1000002a5e6.00000002/head//4, version: 3051'5071, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d6f50db5/1000002a5e6.00000002/head//4@3051'5071, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3931> 2014-11-08 18:15:28.583852 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495bc00 con 0x1c5c0580 -3930> 2014-11-08 18:15:28.583881 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495b660 con 0x1c5c0580 -3929> 2014-11-08 18:15:28.583888 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x24958d20 con 0x1c5c0580 -3928> 2014-11-08 18:15:28.583894 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x249594a0 con 0x1c5c0580 -3927> 2014-11-08 18:15:28.583899 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2495e720 con 0x1c5c0580 -3926> 2014-11-08 18:15:28.583907 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(5f2f8b16/1000002b1f1.0000003c/head//4),PushReplyOp(359f8b16/1000002bdb6.0000007f/head//4),PushReplyOp(9b809b16/1000002a4af.00000000/head//4),PushReplyOp(e5d09b16/10000028aae.0000000e/head//4),PushReplyOp(3319b16/10000019a01.00000033/head//4)]) v2 -- ?+0 0x21617200 con 0x1c5c0580 -3925> 2014-11-08 18:15:28.589522 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8610 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) v3 ==== 757+0+0 (155033801 0 0) 0x226f6780 con 0x1c5c0580 -3924> 2014-11-08 18:15:28.589563 7fc5a152e700 5 -- op tracker -- seq: 47178, time: 2014-11-08 18:15:28.589052, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3923> 2014-11-08 18:15:28.589574 7fc5a152e700 5 -- op tracker -- seq: 47178, time: 2014-11-08 18:15:28.589056, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3922> 2014-11-08 18:15:28.589578 7fc5a152e700 5 -- op tracker -- seq: 47178, time: 2014-11-08 18:15:28.589514, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3921> 2014-11-08 18:15:28.589582 7fc5a152e700 5 -- op tracker -- seq: 47178, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3920> 2014-11-08 18:15:28.589910 7fc5afb15700 5 -- op tracker -- seq: 47178, time: 2014-11-08 18:15:28.589910, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3919> 2014-11-08 18:15:28.589923 7fc5afb15700 5 -- op tracker -- seq: 47178, time: 2014-11-08 18:15:28.589923, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3918> 2014-11-08 18:15:28.589969 7fc5afb15700 5 -- op tracker -- seq: 47178, time: 2014-11-08 18:15:28.589969, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 3319b16/10000019a01.00000033/head//4) -3917> 2014-11-08 18:15:28.616667 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 512 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:28.613362) v2 ==== 47+0+0 (1994781266 0 0) 0x1e6ac560 con 0x1eb9d800 -3916> 2014-11-08 18:15:28.616699 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.613362) v2 -- ?+0 0x1e6ac740 con 0x1eb9d800 -3915> 2014-11-08 18:15:28.617003 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 512 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:28.613362) v2 ==== 47+0+0 (1994781266 0 0) 0x1f4c5640 con 0x1eb9c8e0 -3914> 2014-11-08 18:15:28.617017 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:28.613362) v2 -- ?+0 0x1e6ae360 con 0x1eb9c8e0 -3913> 2014-11-08 18:15:28.708315 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x25fc2760 con 0x1c5c0580 -3912> 2014-11-08 18:15:28.708347 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x25fc2d00 con 0x1c5c0580 -3911> 2014-11-08 18:15:28.708353 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x25fc3480 con 0x1c5c0580 -3910> 2014-11-08 18:15:28.708360 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(2b550db5/1000002b4fc.00000040/head//4),PushReplyOp(ccd50db5/1000002a4e5.00000005/head//4),PushReplyOp(d6f50db5/1000002a5e6.00000002/head//4)]) v2 -- ?+0 0x21612c00 con 0x1c5c0580 -3909> 2014-11-08 18:15:28.711320 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8611 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) v3 ==== 757+0+0 (1243027036 0 0) 0x226f4c80 con 0x1c5c0580 -3908> 2014-11-08 18:15:28.711337 7fc5a152e700 5 -- op tracker -- seq: 47179, time: 2014-11-08 18:15:28.711267, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3907> 2014-11-08 18:15:28.711344 7fc5a152e700 5 -- op tracker -- seq: 47179, time: 2014-11-08 18:15:28.711270, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3906> 2014-11-08 18:15:28.711347 7fc5a152e700 5 -- op tracker -- seq: 47179, time: 2014-11-08 18:15:28.711315, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3905> 2014-11-08 18:15:28.711350 7fc5a152e700 5 -- op tracker -- seq: 47179, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3904> 2014-11-08 18:15:28.711390 7fc5adb11700 5 -- op tracker -- seq: 47179, time: 2014-11-08 18:15:28.711390, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3903> 2014-11-08 18:15:28.711403 7fc5adb11700 5 -- op tracker -- seq: 47179, time: 2014-11-08 18:15:28.711403, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3902> 2014-11-08 18:15:28.711450 7fc5adb11700 5 -- op tracker -- seq: 47179, time: 2014-11-08 18:15:28.711450, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb d6f50db5/1000002a5e6.00000002/head//4) -3901> 2014-11-08 18:15:29.049588 7fc5c362f700 5 osd.9 5773 tick -3900> 2014-11-08 18:15:29.049618 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:09.049615) -3899> 2014-11-08 18:15:29.116288 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 513 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:29.114169) v2 ==== 47+0+0 (2982277528 0 0) 0x1e6aab20 con 0x1eb9d800 -3898> 2014-11-08 18:15:29.116308 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 513 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:29.114169) v2 ==== 47+0+0 (2982277528 0 0) 0x1f4c0f00 con 0x1eb9c8e0 -3897> 2014-11-08 18:15:29.116341 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:29.114169) v2 -- ?+0 0x1e6ac560 con 0x1eb9d800 -3896> 2014-11-08 18:15:29.116376 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:29.114169) v2 -- ?+0 0x1f4c5640 con 0x1eb9c8e0 -3895> 2014-11-08 18:15:29.206596 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8612 ==== MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/1000001b2bc.00000030/ head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8399580+0+0 (4040273386 0 0) 0x21612c00 con 0x1c5c0580 -3894> 2014-11-08 18:15:29.206647 7fc5a152e700 5 -- op tracker -- seq: 47180, time: 2014-11-08 18:15:29.096591, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/ 1000001b2bc.00000030/head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3893> 2014-11-08 18:15:29.206675 7fc5a152e700 5 -- op tracker -- seq: 47180, time: 2014-11-08 18:15:29.096594, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/ 1000001b2bc.00000030/head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3892> 2014-11-08 18:15:29.206709 7fc5a152e700 5 -- op tracker -- seq: 47180, time: 2014-11-08 18:15:29.206588, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/ 1000001b2bc.00000030/head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3891> 2014-11-08 18:15:29.206735 7fc5a152e700 5 -- op tracker -- seq: 47180, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(! first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/1000001b2bc.00000030/ head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3890> 2014-11-08 18:15:29.206793 7fc5ad310700 5 -- op tracker -- seq: 47180, time: 2014-11-08 18:15:29.206793, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/ 1000001b2bc.00000030/head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3889> 2014-11-08 18:15:29.209741 7fc5a152e700 5 -- op tracker -- seq: 47180, time: 2014-11-08 18:15:29.209741, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(83329b16/1000000fb7b.00000000/head//4, version: 397'383, data_included: [0~6144], data_size: 6144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(83329b16/1000000fb7b.00000000/head//4@397'383, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:12288, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(33a29b16/10000029715.0000004b/head//4, version: 3033'4344, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(33a29b16/10000029715.0000004b/head//4@3033'4344, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d8039b16/1000002a72a.00000043/head//4, version: 3458'5308, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d8039b16/1000002a72a.00000043/head//4@3458'5308, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(ed939b16/1000001b2bc.00000030/head//4, version: 2829'2730, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ed939b16/ 1000001b2bc.00000030/head//4@2829'2730, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(17d39b16/1000002a3c8.00000028/head//4, version: 3033'4881, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(17d39b16/1000002a3c8.00000028/head//4@3033'4881, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3888> 2014-11-08 18:15:29.350047 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8613 ==== MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (3414138573 0 0) 0x21612c00 con 0x1c5c0580 -3887> 2014-11-08 18:15:29.350108 7fc5a152e700 5 -- op tracker -- seq: 47181, time: 2014-11-08 18:15:29.239734, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3886> 2014-11-08 18:15:29.350132 7fc5a152e700 5 -- op tracker -- seq: 47181, time: 2014-11-08 18:15:29.239739, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3885> 2014-11-08 18:15:29.350150 7fc5a152e700 5 -- op tracker -- seq: 47181, time: 2014-11-08 18:15:29.350034, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3884> 2014-11-08 18:15:29.350169 7fc5a152e700 5 -- op tracker -- seq: 47181, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3883> 2014-11-08 18:15:29.350575 7fc5b0316700 5 -- op tracker -- seq: 47181, time: 2014-11-08 18:15:29.350575, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3882> 2014-11-08 18:15:29.350711 7fc5b0316700 5 -- op tracker -- seq: 47181, time: 2014-11-08 18:15:29.350711, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(e760db5/1000001b2c0.0000005c/head//4, version: 2829'2832, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(e760db5/1000001b2c0.0000005c/head//4@2829'2832, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(66a60db5/10000019b1c.00000779/head//4, version: 397'1187, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(66a60db5/10000019b1c.00000779/head//4@397'1187, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(da60db5/1000001ac5f.0000008d/head//4, version: 2829'2646, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(da60db5/1000001ac5f.0000008d/head//4@2829'2646, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3881> 2014-11-08 18:15:29.407780 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25fc4ec0 con 0x1c5c0580 -3880> 2014-11-08 18:15:29.407825 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x25fc1c20 con 0x1c5c0580 -3879> 2014-11-08 18:15:29.407832 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1db1d460 con 0x1c5c0580 -3878> 2014-11-08 18:15:29.407838 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1db18960 con 0x1c5c0580 -3877> 2014-11-08 18:15:29.407843 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1db1a760 con 0x1c5c0580 -3876> 2014-11-08 18:15:29.525454 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1db1b840 con 0x1c5c0580 -3875> 2014-11-08 18:15:29.525496 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1db18f00 con 0x1c5c0580 -3874> 2014-11-08 18:15:29.525503 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1db1eea0 con 0x1c5c0580 -3873> 2014-11-08 18:15:29.659565 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3720 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:29.657608) v2 ==== 47+0+0 (111394818 0 0) 0x2515fda0 con 0x1c9f8580 -3872> 2014-11-08 18:15:29.659608 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:29.657608) v2 -- ?+0 0x1e6aab20 con 0x1c9f8580 -3871> 2014-11-08 18:15:29.659926 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3720 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:29.657608) v2 ==== 47+0+0 (111394818 0 0) 0x22d52d00 con 0x1c8b7d20 -3870> 2014-11-08 18:15:29.659946 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:29.657608) v2 -- ?+0 0x1f4c0f00 con 0x1c8b7d20 -3869> 2014-11-08 18:15:29.788712 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(83329b16/1000000fb7b.00000000/head//4),PushReplyOp(33a29b16/10000029715.0000004b/head//4),PushReplyOp(d8039b16/1000002a72a.00000043/head//4),PushReplyOp(ed939b16/1000001b2bc.00000030/head//4),PushReplyOp(17d39b16/1000002a3c8.00000028/head//4)]) v2 -- ?+0 0x1c6a7c00 con 0x1c5c0580 -3868> 2014-11-08 18:15:29.795565 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8614 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) v3 ==== 757+0+0 (3468864903 0 0) 0x226f0d80 con 0x1c5c0580 -3867> 2014-11-08 18:15:29.795602 7fc5a152e700 5 -- op tracker -- seq: 47182, time: 2014-11-08 18:15:29.795462, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3866> 2014-11-08 18:15:29.795614 7fc5a152e700 5 -- op tracker -- seq: 47182, time: 2014-11-08 18:15:29.795466, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3865> 2014-11-08 18:15:29.795617 7fc5a152e700 5 -- op tracker -- seq: 47182, time: 2014-11-08 18:15:29.795555, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3864> 2014-11-08 18:15:29.795620 7fc5a152e700 5 -- op tracker -- seq: 47182, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3863> 2014-11-08 18:15:29.795658 7fc5afb15700 5 -- op tracker -- seq: 47182, time: 2014-11-08 18:15:29.795658, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3862> 2014-11-08 18:15:29.795671 7fc5afb15700 5 -- op tracker -- seq: 47182, time: 2014-11-08 18:15:29.795671, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3861> 2014-11-08 18:15:29.795851 7fc5afb15700 5 -- op tracker -- seq: 47182, time: 2014-11-08 18:15:29.795850, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 17d39b16/1000002a3c8.00000028/head//4) -3860> 2014-11-08 18:15:29.807949 7fc5bf627700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(e760db5/1000001b2c0.0000005c/head//4),PushReplyOp(66a60db5/10000019b1c.00000779/head//4),PushReplyOp(da60db5/1000001ac5f.0000008d/head//4)]) v2 -- ?+0 0x256ffc00 con 0x1c5c0580 -3859> 2014-11-08 18:15:29.822130 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8615 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) v3 ==== 757+0+0 (1592603309 0 0) 0x226f7080 con 0x1c5c0580 -3858> 2014-11-08 18:15:29.822166 7fc5a152e700 5 -- op tracker -- seq: 47183, time: 2014-11-08 18:15:29.822035, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3857> 2014-11-08 18:15:29.822176 7fc5a152e700 5 -- op tracker -- seq: 47183, time: 2014-11-08 18:15:29.822039, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3856> 2014-11-08 18:15:29.822180 7fc5a152e700 5 -- op tracker -- seq: 47183, time: 2014-11-08 18:15:29.822121, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3855> 2014-11-08 18:15:29.822185 7fc5a152e700 5 -- op tracker -- seq: 47183, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3854> 2014-11-08 18:15:29.822224 7fc5adb11700 5 -- op tracker -- seq: 47183, time: 2014-11-08 18:15:29.822224, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3853> 2014-11-08 18:15:29.822237 7fc5adb11700 5 -- op tracker -- seq: 47183, time: 2014-11-08 18:15:29.822236, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3852> 2014-11-08 18:15:29.822285 7fc5adb11700 5 -- op tracker -- seq: 47183, time: 2014-11-08 18:15:29.822285, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb da60db5/1000001ac5f.0000008d/head//4) -3851> 2014-11-08 18:15:29.922549 7fc5b2b1b700 10 monclient: tick -3850> 2014-11-08 18:15:29.922576 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:14:59.922575) -3849> 2014-11-08 18:15:29.922589 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:15:29.922589; renew after: 2014-11-08 18:16:29.290049) -- no -3848> 2014-11-08 18:15:30.049733 7fc5c362f700 5 osd.9 5773 tick -3847> 2014-11-08 18:15:30.049760 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:10.049757) -3846> 2014-11-08 18:15:30.073193 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2295 ==== MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (2067159275 0 0) 0x256fdc00 con 0x1e50aec0 -3845> 2014-11-08 18:15:30.073230 7fc5a56ef700 5 -- op tracker -- seq: 47184, time: 2014-11-08 18:15:30.050013, event: header_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3844> 2014-11-08 18:15:30.073243 7fc5a56ef700 5 -- op tracker -- seq: 47184, time: 2014-11-08 18:15:30.050016, event: throttled, op: MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3843> 2014-11-08 18:15:30.073251 7fc5a56ef700 5 -- op tracker -- seq: 47184, time: 2014-11-08 18:15:30.073166, event: all_read, op: MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3842> 2014-11-08 18:15:30.073259 7fc5a56ef700 5 -- op tracker -- seq: 47184, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3841> 2014-11-08 18:15:30.073324 7fc5af314700 5 -- op tracker -- seq: 47184, time: 2014-11-08 18:15:30.073324, event: reached_pg, op: MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3840> 2014-11-08 18:15:30.077719 7fc5a56ef700 5 -- op tracker -- seq: 47184, time: 2014-11-08 18:15:30.077719, event: done, op: MOSDPGPush(4.b7s1 5773 [PushOp(a270a0b7/1000001987b.000000b3/head//4, version: 397'859, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a270a0b7/1000001987b.000000b3/head//4@397'859, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3839> 2014-11-08 18:15:30.115729 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_trim(4.b7s1 to 5734'8245 e5773) v1 -- ?+0 0x1db1f260 con 0x1e50aec0 -3838> 2014-11-08 18:15:30.115838 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPushReply(4.b7s1 5773 [PushReplyOp(a270a0b7/1000001987b.000000b3/head//4)]) v2 -- ?+0 0x21611200 con 0x1e50aec0 -3837> 2014-11-08 18:15:30.160331 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3721 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:30.158377) v2 ==== 47+0+0 (866191559 0 0) 0x22d55a00 con 0x1c8b7d20 -3836> 2014-11-08 18:15:30.160383 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:30.158377) v2 -- ?+0 0x22d52d00 con 0x1c8b7d20 -3835> 2014-11-08 18:15:30.160431 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3721 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:30.158377) v2 ==== 47+0+0 (866191559 0 0) 0x257c7440 con 0x1c9f8580 -3834> 2014-11-08 18:15:30.160443 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:30.158377) v2 -- ?+0 0x2515fda0 con 0x1c9f8580 -3833> 2014-11-08 18:15:30.465408 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8616 ==== MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (618290052 0 0) 0x256ffc00 con 0x1c5c0580 -3832> 2014-11-08 18:15:30.465457 7fc5a152e700 5 -- op tracker -- seq: 47185, time: 2014-11-08 18:15:30.387785, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3831> 2014-11-08 18:15:30.465474 7fc5a152e700 5 -- op tracker -- seq: 47185, time: 2014-11-08 18:15:30.387793, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3830> 2014-11-08 18:15:30.465486 7fc5a152e700 5 -- op tracker -- seq: 47185, time: 2014-11-08 18:15:30.465374, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3829> 2014-11-08 18:15:30.465497 7fc5a152e700 5 -- op tracker -- seq: 47185, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3828> 2014-11-08 18:15:30.465542 7fc5b0316700 5 -- op tracker -- seq: 47185, time: 2014-11-08 18:15:30.465542, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3827> 2014-11-08 18:15:30.480415 7fc5a152e700 5 -- op tracker -- seq: 47185, time: 2014-11-08 18:15:30.480415, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(1ac60db5/1000002b128.00000233/head//4, version: 4157'7898, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1ac60db5/1000002b128.00000233/head//4@4157'7898, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6e570db5/1000002bc41.00000081/head//4, version: 3478'7434, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6e570db5/1000002bc41.00000081/head//4@3478'7434, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(8d970db5/1000002a3af.00000012/head//4, version: 3033'4812, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(8d970db5/1000002a3af.00000012/head//4@3033'4812, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3826> 2014-11-08 18:15:30.548229 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1db1cb00 con 0x1c5c0580 -3825> 2014-11-08 18:15:30.548265 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1db185a0 con 0x1c5c0580 -3824> 2014-11-08 18:15:30.548274 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x21227bc0 con 0x1c5c0580 -3823> 2014-11-08 18:15:30.548284 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(1ac60db5/1000002b128.00000233/head//4),PushReplyOp(6e570db5/1000002bc41.00000081/head//4),PushReplyOp(8d970db5/1000002a3af.00000012/head//4)]) v2 -- ?+0 0x21612c00 con 0x1c5c0580 -3822> 2014-11-08 18:15:30.642990 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8617 ==== MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d59b16/ 10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (847121510 0 0) 0x256ffc00 con 0x1c5c0580 -3821> 2014-11-08 18:15:30.643054 7fc5a152e700 5 -- op tracker -- seq: 47186, time: 2014-11-08 18:15:30.537845, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( c8d59b16/10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3820> 2014-11-08 18:15:30.643100 7fc5a152e700 5 -- op tracker -- seq: 47186, time: 2014-11-08 18:15:30.537850, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d59b16/ 10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3819> 2014-11-08 18:15:30.643125 7fc5a152e700 5 -- op tracker -- seq: 47186, time: 2014-11-08 18:15:30.642979, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d59b16/ 10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3818> 2014-11-08 18:15:30.643150 7fc5a152e700 5 -- op tracker -- seq: 47186, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d59b16/ 10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3817> 2014-11-08 18:15:30.643208 7fc5ad310700 5 -- op tracker -- seq: 47186, time: 2014-11-08 18:15:30.643208, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d59b16/ 10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3816> 2014-11-08 18:15:30.656630 7fc5a152e700 5 -- op tracker -- seq: 47186, time: 2014-11-08 18:15:30.656630, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(fb549b16/1000002a451.000000ab/head//4, version: 3033'4933, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(fb549b16/1000002a451.000000ab/head//4@3033'4933, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(83649b16/10000019746.0000004f/head//4, version: 397'737, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(83649b16/10000019746.0000004f/head//4@397'737, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(96849b16/1000002b58a.00000069/head//4, version: 3473'6629, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(96849b16/1000002b58a.00000069/head//4@3473'6629, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8d59b16/10000028516.0000000d/head//4, version: 3033'3785, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8d59b16/ 10000028516.0000000d/head//4@3033'3785, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f2269b16/1000002a7a2.00000056/head//4, version: 3458'5424, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f2269b16/1000002a7a2.00000056/head//4@3458'5424, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3815> 2014-11-08 18:15:30.656882 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8618 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) v3 ==== 757+0+0 (850602974 0 0) 0x22767500 con 0x1c5c0580 -3814> 2014-11-08 18:15:30.656904 7fc5a152e700 5 -- op tracker -- seq: 47187, time: 2014-11-08 18:15:30.656785, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3813> 2014-11-08 18:15:30.656914 7fc5a152e700 5 -- op tracker -- seq: 47187, time: 2014-11-08 18:15:30.656787, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3812> 2014-11-08 18:15:30.656924 7fc5a152e700 5 -- op tracker -- seq: 47187, time: 2014-11-08 18:15:30.656875, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3811> 2014-11-08 18:15:30.656934 7fc5a152e700 5 -- op tracker -- seq: 47187, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3810> 2014-11-08 18:15:30.656979 7fc5adb11700 5 -- op tracker -- seq: 47187, time: 2014-11-08 18:15:30.656979, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3809> 2014-11-08 18:15:30.656989 7fc5adb11700 5 -- op tracker -- seq: 47187, time: 2014-11-08 18:15:30.656989, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3808> 2014-11-08 18:15:30.657046 7fc5a152e700 5 -- op tracker -- seq: 47187, time: 2014-11-08 18:15:30.657046, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 8d970db5/1000002a3af.00000012/head//4) -3807> 2014-11-08 18:15:30.707934 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8619 ==== MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (140969680 0 0) 0x21612c00 con 0x1c5c0580 -3806> 2014-11-08 18:15:30.707976 7fc5a152e700 5 -- op tracker -- seq: 47188, time: 2014-11-08 18:15:30.680726, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3805> 2014-11-08 18:15:30.707989 7fc5a152e700 5 -- op tracker -- seq: 47188, time: 2014-11-08 18:15:30.680731, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3804> 2014-11-08 18:15:30.707997 7fc5a152e700 5 -- op tracker -- seq: 47188, time: 2014-11-08 18:15:30.707923, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3803> 2014-11-08 18:15:30.708005 7fc5a152e700 5 -- op tracker -- seq: 47188, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3802> 2014-11-08 18:15:30.708048 7fc5b0316700 5 -- op tracker -- seq: 47188, time: 2014-11-08 18:15:30.708048, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3801> 2014-11-08 18:15:30.708136 7fc5a152e700 5 -- op tracker -- seq: 47188, time: 2014-11-08 18:15:30.708136, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(11680db5/10000019b35.00000619/head//4, version: 397'1228, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(11680db5/10000019b35.00000619/head//4@397'1228, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3800> 2014-11-08 18:15:30.783531 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x21225820 con 0x1c5c0580 -3799> 2014-11-08 18:15:30.783580 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x21223c00 con 0x1c5c0580 -3798> 2014-11-08 18:15:30.783587 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x21224b00 con 0x1c5c0580 -3797> 2014-11-08 18:15:30.783592 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2336d0a0 con 0x1c5c0580 -3796> 2014-11-08 18:15:30.783665 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2336f620 con 0x1c5c0580 -3795> 2014-11-08 18:15:30.783675 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(fb549b16/1000002a451.000000ab/head//4),PushReplyOp(83649b16/10000019746.0000004f/head//4),PushReplyOp(96849b16/1000002b58a.00000069/head//4),PushReplyOp(c8d59b16/10000028516.0000000d/head//4),PushReplyOp(f2269b16/1000002a7a2.00000056/head//4)]) v2 -- ?+0 0x229b9600 con 0x1c5c0580 -3794> 2014-11-08 18:15:30.786389 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8620 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) v3 ==== 757+0+0 (219198469 0 0) 0x22767500 con 0x1c5c0580 -3793> 2014-11-08 18:15:30.786420 7fc5a152e700 5 -- op tracker -- seq: 47189, time: 2014-11-08 18:15:30.786292, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3792> 2014-11-08 18:15:30.786429 7fc5a152e700 5 -- op tracker -- seq: 47189, time: 2014-11-08 18:15:30.786295, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3791> 2014-11-08 18:15:30.786433 7fc5a152e700 5 -- op tracker -- seq: 47189, time: 2014-11-08 18:15:30.786380, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3790> 2014-11-08 18:15:30.786435 7fc5a152e700 5 -- op tracker -- seq: 47189, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3789> 2014-11-08 18:15:30.786653 7fc5afb15700 5 -- op tracker -- seq: 47189, time: 2014-11-08 18:15:30.786653, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3788> 2014-11-08 18:15:30.786665 7fc5afb15700 5 -- op tracker -- seq: 47189, time: 2014-11-08 18:15:30.786665, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3787> 2014-11-08 18:15:30.786718 7fc5afb15700 5 -- op tracker -- seq: 47189, time: 2014-11-08 18:15:30.786718, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb f2269b16/1000002a7a2.00000056/head//4) -3786> 2014-11-08 18:15:30.835363 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2336b480 con 0x1c5c0580 -3785> 2014-11-08 18:15:30.835405 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(11680db5/10000019b35.00000619/head//4)]) v2 -- ?+0 0x1d85ae00 con 0x1c5c0580 -3784> 2014-11-08 18:15:30.838173 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8621 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) v3 ==== 757+0+0 (1781790688 0 0) 0x22766c00 con 0x1c5c0580 -3783> 2014-11-08 18:15:30.838192 7fc5a152e700 5 -- op tracker -- seq: 47190, time: 2014-11-08 18:15:30.838109, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3782> 2014-11-08 18:15:30.838202 7fc5a152e700 5 -- op tracker -- seq: 47190, time: 2014-11-08 18:15:30.838111, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3781> 2014-11-08 18:15:30.838207 7fc5a152e700 5 -- op tracker -- seq: 47190, time: 2014-11-08 18:15:30.838158, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3780> 2014-11-08 18:15:30.838211 7fc5a152e700 5 -- op tracker -- seq: 47190, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3779> 2014-11-08 18:15:30.838243 7fc5adb11700 5 -- op tracker -- seq: 47190, time: 2014-11-08 18:15:30.838243, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3778> 2014-11-08 18:15:30.838253 7fc5adb11700 5 -- op tracker -- seq: 47190, time: 2014-11-08 18:15:30.838253, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3777> 2014-11-08 18:15:30.838297 7fc5adb11700 5 -- op tracker -- seq: 47190, time: 2014-11-08 18:15:30.838297, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 11680db5/10000019b35.00000619/head//4) -3776> 2014-11-08 18:15:31.049897 7fc5c362f700 5 osd.9 5773 tick -3775> 2014-11-08 18:15:31.049920 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:11.049917) -3774> 2014-11-08 18:15:31.049938 7fc5c362f700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -3773> 2014-11-08 18:15:31.049945 7fc5c362f700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- osd_failure(failed osd.4 192.168.1.16:6805/31236 for 26sec e5773 v5773) v3 -- ?+0 0x2183da00 con 0x44ed6a0 -3772> 2014-11-08 18:15:31.219312 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12069 ==== MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) v1 ==== 298+0+0 (1765060254 0 0) 0x1ce618c0 con 0x1c5c2680 -3771> 2014-11-08 18:15:31.219335 7fc5a6cfe700 5 -- op tracker -- seq: 47191, time: 2014-11-08 18:15:31.219148, event: header_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) -3770> 2014-11-08 18:15:31.219342 7fc5a6cfe700 5 -- op tracker -- seq: 47191, time: 2014-11-08 18:15:31.219151, event: throttled, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) -3769> 2014-11-08 18:15:31.219347 7fc5a6cfe700 5 -- op tracker -- seq: 47191, time: 2014-11-08 18:15:31.219305, event: all_read, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) -3768> 2014-11-08 18:15:31.219351 7fc5a6cfe700 5 -- op tracker -- seq: 47191, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) -3767> 2014-11-08 18:15:31.219382 7fc5ad310700 5 -- op tracker -- seq: 47191, time: 2014-11-08 18:15:31.219382, event: reached_pg, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) -3766> 2014-11-08 18:15:31.289302 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8622 ==== MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (485834426 0 0) 0x1d85ae00 con 0x1c5c0580 -3765> 2014-11-08 18:15:31.289353 7fc5a152e700 5 -- op tracker -- seq: 47192, time: 2014-11-08 18:15:31.209583, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3764> 2014-11-08 18:15:31.289372 7fc5a152e700 5 -- op tracker -- seq: 47192, time: 2014-11-08 18:15:31.209587, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3763> 2014-11-08 18:15:31.289385 7fc5a152e700 5 -- op tracker -- seq: 47192, time: 2014-11-08 18:15:31.289289, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3762> 2014-11-08 18:15:31.289427 7fc5a152e700 5 -- op tracker -- seq: 47192, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3761> 2014-11-08 18:15:31.289474 7fc5afb15700 5 -- op tracker -- seq: 47192, time: 2014-11-08 18:15:31.289474, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3760> 2014-11-08 18:15:31.298151 7fc5a152e700 5 -- op tracker -- seq: 47192, time: 2014-11-08 18:15:31.298150, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(26a69b16/1000002ae36.000002c6/head//4, version: 3468'6155, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(26a69b16/1000002ae36.000002c6/head//4@3468'6155, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1979b16/1000002adaf.00000049/head//4, version: 3468'6098, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1979b16/1000002adaf.00000049/head//4@3468'6098, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2589b16/1000002ab99.0000002f/head//4, version: 3468'5871, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2589b16/1000002ab99.0000002f/head//4@3468'5871, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3759> 2014-11-08 18:15:31.319860 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3687 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:31.316208) v2 ==== 47+0+0 (1524688031 0 0) 0x1caab660 con 0x1cce2ec0 -3758> 2014-11-08 18:15:31.319888 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:31.316208) v2 -- ?+0 0x257c7440 con 0x1cce2ec0 -3757> 2014-11-08 18:15:31.319907 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3687 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:31.316208) v2 ==== 47+0+0 (1524688031 0 0) 0x23680000 con 0x1cce3440 -3756> 2014-11-08 18:15:31.319916 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:31.316208) v2 -- ?+0 0x22d55a00 con 0x1cce3440 -3755> 2014-11-08 18:15:31.417058 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 514 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:31.414937) v2 ==== 47+0+0 (4178748995 0 0) 0x1e6ab480 con 0x1eb9d800 -3754> 2014-11-08 18:15:31.417099 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:31.414937) v2 -- ?+0 0x1caab660 con 0x1eb9d800 -3753> 2014-11-08 18:15:31.417933 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 514 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:31.414937) v2 ==== 47+0+0 (4178748995 0 0) 0x1f4c3de0 con 0x1eb9c8e0 -3752> 2014-11-08 18:15:31.417947 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:31.414937) v2 -- ?+0 0x23680000 con 0x1eb9c8e0 -3751> 2014-11-08 18:15:31.520856 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2336c740 con 0x1c5c0580 -3750> 2014-11-08 18:15:31.520915 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x22868d20 con 0x1c5c0580 -3749> 2014-11-08 18:15:31.520923 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x2286d640 con 0x1c5c0580 -3748> 2014-11-08 18:15:31.520932 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(26a69b16/1000002ae36.000002c6/head//4),PushReplyOp(c1979b16/1000002adaf.00000049/head//4),PushReplyOp(2589b16/1000002ab99.0000002f/head//4)]) v2 -- ?+0 0x1d3a8e00 con 0x1c5c0580 -3747> 2014-11-08 18:15:31.525878 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8623 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) v3 ==== 757+0+0 (2216194546 0 0) 0x248b5e80 con 0x1c5c0580 -3746> 2014-11-08 18:15:31.525894 7fc5a152e700 5 -- op tracker -- seq: 47193, time: 2014-11-08 18:15:31.525828, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3745> 2014-11-08 18:15:31.525903 7fc5a152e700 5 -- op tracker -- seq: 47193, time: 2014-11-08 18:15:31.525830, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3744> 2014-11-08 18:15:31.525906 7fc5a152e700 5 -- op tracker -- seq: 47193, time: 2014-11-08 18:15:31.525872, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3743> 2014-11-08 18:15:31.525909 7fc5a152e700 5 -- op tracker -- seq: 47193, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3742> 2014-11-08 18:15:31.525946 7fc5afb15700 5 -- op tracker -- seq: 47193, time: 2014-11-08 18:15:31.525946, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3741> 2014-11-08 18:15:31.525954 7fc5afb15700 5 -- op tracker -- seq: 47193, time: 2014-11-08 18:15:31.525954, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3740> 2014-11-08 18:15:31.526175 7fc5a152e700 5 -- op tracker -- seq: 47193, time: 2014-11-08 18:15:31.526175, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 2589b16/1000002ab99.0000002f/head//4) -3739> 2014-11-08 18:15:31.918210 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 515 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:31.915906) v2 ==== 47+0+0 (682257506 0 0) 0x1e6ab660 con 0x1eb9d800 -3738> 2014-11-08 18:15:31.918242 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:31.915906) v2 -- ?+0 0x1e6ab480 con 0x1eb9d800 -3737> 2014-11-08 18:15:31.918323 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 515 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:31.915906) v2 ==== 47+0+0 (682257506 0 0) 0x2127b480 con 0x1eb9c8e0 -3736> 2014-11-08 18:15:31.918335 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:31.915906) v2 -- ?+0 0x1f4c3de0 con 0x1eb9c8e0 -3735> 2014-11-08 18:15:32.050045 7fc5c362f700 5 osd.9 5773 tick -3734> 2014-11-08 18:15:32.050078 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:12.050075) -3733> 2014-11-08 18:15:32.225509 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MOSDECSubOpReadReply(4.bbs0 5773 ECSubReadReply(tid=25202, attrs_read=0)) v1 -- ?+0 0x21a4c880 con 0x1c5c2680 -3732> 2014-11-08 18:15:32.229730 7fc5ad310700 5 -- op tracker -- seq: 47191, time: 2014-11-08 18:15:32.229730, event: done, op: MOSDECSubOpRead(4.bbs2 5773 ECSubRead(tid=25202, to_read={d453a0bb/1000001a239.000002ca/head//4=0,4194304,8d05a0bb/1000002b688.00000078/head//4=0,4194304,5c25a0bb/1000001a021.00000a93/head//4=0,4194304}, attrs_to_read=)) -3731> 2014-11-08 18:15:32.263969 7fc5aa30a700 5 osd.9 5773 heartbeat: osd_stat(1712 GB used, 2012 GB avail, 3726 GB total, peers [0,1,2,3,4,5,8,10,11]/[] op hist []) -3730> 2014-11-08 18:15:32.264012 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6802/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1d636ea0 con 0x1cae2940 -3729> 2014-11-08 18:15:32.264038 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6803/1266 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1d6305a0 con 0x1cae2680 -3728> 2014-11-08 18:15:32.264049 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6807/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x2583f080 con 0x1c5c02c0 -3727> 2014-11-08 18:15:32.264060 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6808/1411 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x23684560 con 0x1cae0160 -3726> 2014-11-08 18:15:32.264070 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6812/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1c830f00 con 0x1cae0f20 -3725> 2014-11-08 18:15:32.264081 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.15:6813/1606 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x2290fda0 con 0x1cae0c60 -3724> 2014-11-08 18:15:32.264091 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6802/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1e5314a0 con 0x1cae23c0 -3723> 2014-11-08 18:15:32.264101 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6803/1238 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x25ed2760 con 0x1cae56a0 -3722> 2014-11-08 18:15:32.264112 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6811/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1eaccce0 con 0x1cbf9ce0 -3721> 2014-11-08 18:15:32.264122 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6812/5031236 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1e6aa760 con 0x1d603020 -3720> 2014-11-08 18:15:32.264132 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6807/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x227e2940 con 0x1dbd9b80 -3719> 2014-11-08 18:15:32.264142 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.16:6808/12001550 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x259f5820 con 0x1d235d80 -3718> 2014-11-08 18:15:32.264152 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6807/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x24959680 con 0x1cae1fa0 -3717> 2014-11-08 18:15:32.264164 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.17:6808/32391 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1d5e4ce0 con 0x1cae1ce0 -3716> 2014-11-08 18:15:32.264174 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6803/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x2127f080 con 0x1cae1340 -3715> 2014-11-08 18:15:32.264184 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6804/1404 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1d1b5460 con 0x1cae1080 -3714> 2014-11-08 18:15:32.264194 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6808/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x227e6900 con 0x1cae1a20 -3713> 2014-11-08 18:15:32.264203 7fc5aa30a700 1 -- 192.168.1.18:0/4928 --> 192.168.1.18:6809/3588 -- osd_ping(ping e5773 stamp 2014-11-08 18:15:32.264009) v2 -- ?+0 0x1eacb840 con 0x1cae2c00 -3712> 2014-11-08 18:15:32.264214 7fc5aa30a700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:12.264212) -3711> 2014-11-08 18:15:32.264993 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6804/1404 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1c831fe0 con 0x1cae1080 -3710> 2014-11-08 18:15:32.265156 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.10 192.168.1.18:6803/1404 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x2583cb00 con 0x1cae1340 -3709> 2014-11-08 18:15:32.265193 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6808/3588 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x23d73a20 con 0x1cae1a20 -3708> 2014-11-08 18:15:32.265230 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.11 192.168.1.18:6809/3588 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1dbfba20 con 0x1cae2c00 -3707> 2014-11-08 18:15:32.267427 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6802/1238 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x22309860 con 0x1cae23c0 -3706> 2014-11-08 18:15:32.268171 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.3 192.168.1.16:6803/1238 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x24885dc0 con 0x1cae56a0 -3705> 2014-11-08 18:15:32.268212 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6807/12001550 329 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x227c9680 con 0x1dbd9b80 -3704> 2014-11-08 18:15:32.268223 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6807/32391 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1eacd0a0 con 0x1cae1fa0 -3703> 2014-11-08 18:15:32.268233 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.8 192.168.1.17:6808/32391 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1ddcab20 con 0x1cae1ce0 -3702> 2014-11-08 18:15:32.268241 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6802/1266 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1dece720 con 0x1cae2940 -3701> 2014-11-08 18:15:32.268251 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.0 192.168.1.15:6803/1266 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1eae23a0 con 0x1cae2680 -3700> 2014-11-08 18:15:32.268293 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.5 192.168.1.16:6808/12001550 329 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1eae0000 con 0x1d235d80 -3699> 2014-11-08 18:15:32.268306 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6807/1411 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1caaddc0 con 0x1c5c02c0 -3698> 2014-11-08 18:15:32.268339 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.1 192.168.1.15:6808/1411 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x1d045820 con 0x1cae0160 -3697> 2014-11-08 18:15:32.268579 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6813/1606 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x4533fc0 con 0x1cae0c60 -3696> 2014-11-08 18:15:32.268595 7fc5b8326700 1 -- 192.168.1.18:0/4928 <== osd.2 192.168.1.15:6812/1606 3742 ==== osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.264009) v2 ==== 47+0+0 (2640175555 0 0) 0x213b9860 con 0x1cae0f20 -3695> 2014-11-08 18:15:32.328647 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3751 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:32.327223) v2 ==== 47+0+0 (1309394062 0 0) 0x2234eae0 con 0x1cce3180 -3694> 2014-11-08 18:15:32.328675 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.327223) v2 -- ?+0 0x1e6ab660 con 0x1cce3180 -3693> 2014-11-08 18:15:32.328705 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3751 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:32.327223) v2 ==== 47+0+0 (1309394062 0 0) 0x2127ecc0 con 0x1cce3700 -3692> 2014-11-08 18:15:32.328715 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.327223) v2 -- ?+0 0x2127b480 con 0x1cce3700 -3691> 2014-11-08 18:15:32.355098 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8624 ==== MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c90db5/ 1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (2608138395 0 0) 0x1d3a8e00 con 0x1c5c0580 -3690> 2014-11-08 18:15:32.355156 7fc5a152e700 5 -- op tracker -- seq: 47194, time: 2014-11-08 18:15:31.869212, event: header_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( d3c90db5/1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3689> 2014-11-08 18:15:32.355181 7fc5a152e700 5 -- op tracker -- seq: 47194, time: 2014-11-08 18:15:31.869217, event: throttled, op: MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( d3c90db5/1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3688> 2014-11-08 18:15:32.355202 7fc5a152e700 5 -- op tracker -- seq: 47194, time: 2014-11-08 18:15:32.355085, event: all_read, op: MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( d3c90db5/1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3687> 2014-11-08 18:15:32.355221 7fc5a152e700 5 -- op tracker -- seq: 47194, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c90db5/ 1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3686> 2014-11-08 18:15:32.355275 7fc5b0316700 5 -- op tracker -- seq: 47194, time: 2014-11-08 18:15:32.355275, event: reached_pg, op: MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( d3c90db5/1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3685> 2014-11-08 18:15:32.355458 7fc5b0316700 5 -- op tracker -- seq: 47194, time: 2014-11-08 18:15:32.355458, event: done, op: MOSDPGPush(4.b5s1 5773 [PushOp(88880db5/1000002bca4.000000a6/head//4, version: 3485'7491, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(88880db5/1000002bca4.000000a6/head//4@3485'7491, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f6290db5/1000002b83a.0000000f/head//4, version: 3473'6943, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f6290db5/1000002b83a.0000000f/head//4@3473'6943, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(2a590db5/1000000e4ca.00000006/head//4, version: 397'396, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2a590db5/1000000e4ca.00000006/head//4@397'396, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d3c90db5/1000001b290.0000006a/head//4, version: 2829'2814, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d3c90db5/ 1000001b290.0000006a/head//4@2829'2814, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(70f90db5/1000002b0b9.00000062/head//4, version: 4157'7824, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(70f90db5/1000002b0b9.00000062/head//4@4157'7824, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3684> 2014-11-08 18:15:32.533767 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2286e540 con 0x1c5c0580 -3683> 2014-11-08 18:15:32.533803 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x2286ddc0 con 0x1c5c0580 -3682> 2014-11-08 18:15:32.533880 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1f4c2ee0 con 0x1c5c0580 -3681> 2014-11-08 18:15:32.533931 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1f4c2b20 con 0x1c5c0580 -3680> 2014-11-08 18:15:32.533944 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.b5s1 to 5734'8276 e5773) v1 -- ?+0 0x1f4c6180 con 0x1c5c0580 -3679> 2014-11-08 18:15:32.533953 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.b5s1 5773 [PushReplyOp(88880db5/1000002bca4.000000a6/head//4),PushReplyOp(f6290db5/1000002b83a.0000000f/head//4),PushReplyOp(2a590db5/1000000e4ca.00000006/head//4),PushReplyOp(d3c90db5/1000001b290.0000006a/head//4),PushReplyOp(70f90db5/1000002b0b9.00000062/head//4)]) v2 -- ?+0 0x24841400 con 0x1c5c0580 -3678> 2014-11-08 18:15:32.543876 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8625 ==== MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (2849684400 0 0) 0x1d85ae00 con 0x1c5c0580 -3677> 2014-11-08 18:15:32.543918 7fc5a152e700 5 -- op tracker -- seq: 47195, time: 2014-11-08 18:15:32.403073, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3676> 2014-11-08 18:15:32.543935 7fc5a152e700 5 -- op tracker -- seq: 47195, time: 2014-11-08 18:15:32.403078, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3675> 2014-11-08 18:15:32.543949 7fc5a152e700 5 -- op tracker -- seq: 47195, time: 2014-11-08 18:15:32.543867, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3674> 2014-11-08 18:15:32.543962 7fc5a152e700 5 -- op tracker -- seq: 47195, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3673> 2014-11-08 18:15:32.544056 7fc5afb15700 5 -- op tracker -- seq: 47195, time: 2014-11-08 18:15:32.544056, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3672> 2014-11-08 18:15:32.548967 7fc5a152e700 5 -- op tracker -- seq: 47195, time: 2014-11-08 18:15:32.548966, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(43e89b16/1000002bb46.0000002e/head//4, version: 3473'7458, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(43e89b16/1000002bb46.0000002e/head//4@3473'7458, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(be299b16/1000001aac7.00000046/head//4, version: 2829'2426, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(be299b16/1000001aac7.00000046/head//4@2829'2426, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(37699b16/1000002b8d7.00000081/head//4, version: 3473'7102, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(37699b16/1000002b8d7.00000081/head//4@3473'7102, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3671> 2014-11-08 18:15:32.549165 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8626 ==== pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) v3 ==== 757+0+0 (335483146 0 0) 0x248b5e80 con 0x1c5c0580 -3670> 2014-11-08 18:15:32.549182 7fc5a152e700 5 -- op tracker -- seq: 47196, time: 2014-11-08 18:15:32.549092, event: header_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3669> 2014-11-08 18:15:32.549188 7fc5a152e700 5 -- op tracker -- seq: 47196, time: 2014-11-08 18:15:32.549093, event: throttled, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3668> 2014-11-08 18:15:32.549193 7fc5a152e700 5 -- op tracker -- seq: 47196, time: 2014-11-08 18:15:32.549160, event: all_read, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3667> 2014-11-08 18:15:32.549197 7fc5a152e700 5 -- op tracker -- seq: 47196, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3666> 2014-11-08 18:15:32.549233 7fc5adb11700 5 -- op tracker -- seq: 47196, time: 2014-11-08 18:15:32.549233, event: reached_pg, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3665> 2014-11-08 18:15:32.549247 7fc5adb11700 5 -- op tracker -- seq: 47196, time: 2014-11-08 18:15:32.549246, event: started, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3664> 2014-11-08 18:15:32.549291 7fc5adb11700 5 -- op tracker -- seq: 47196, time: 2014-11-08 18:15:32.549291, event: done, op: pg_backfill(progress 4.b5s1 e 5773/5773 lb 70f90db5/1000002b0b9.00000062/head//4) -3663> 2014-11-08 18:15:32.650080 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c30c0 con 0x1c5c0580 -3662> 2014-11-08 18:15:32.650109 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c4b00 con 0x1c5c0580 -3661> 2014-11-08 18:15:32.650116 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c2580 con 0x1c5c0580 -3660> 2014-11-08 18:15:32.650124 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(43e89b16/1000002bb46.0000002e/head//4),PushReplyOp(be299b16/1000001aac7.00000046/head//4),PushReplyOp(37699b16/1000002b8d7.00000081/head//4)]) v2 -- ?+0 0x1d3af400 con 0x1c5c0580 -3659> 2014-11-08 18:15:32.665184 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8627 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) v3 ==== 757+0+0 (1641681635 0 0) 0x1d3cb180 con 0x1c5c0580 -3658> 2014-11-08 18:15:32.665222 7fc5a152e700 5 -- op tracker -- seq: 47197, time: 2014-11-08 18:15:32.665091, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3657> 2014-11-08 18:15:32.665232 7fc5a152e700 5 -- op tracker -- seq: 47197, time: 2014-11-08 18:15:32.665095, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3656> 2014-11-08 18:15:32.665236 7fc5a152e700 5 -- op tracker -- seq: 47197, time: 2014-11-08 18:15:32.665175, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3655> 2014-11-08 18:15:32.665239 7fc5a152e700 5 -- op tracker -- seq: 47197, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3654> 2014-11-08 18:15:32.665680 7fc5ad310700 5 -- op tracker -- seq: 47197, time: 2014-11-08 18:15:32.665680, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3653> 2014-11-08 18:15:32.665699 7fc5ad310700 5 -- op tracker -- seq: 47197, time: 2014-11-08 18:15:32.665699, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3652> 2014-11-08 18:15:32.665750 7fc5ad310700 5 -- op tracker -- seq: 47197, time: 2014-11-08 18:15:32.665750, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 37699b16/1000002b8d7.00000081/head//4) -3651> 2014-11-08 18:15:32.803777 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 326 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:32.801041) v2 ==== 47+0+0 (4092895305 0 0) 0x224a5820 con 0x1eb9c780 -3650> 2014-11-08 18:15:32.803800 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 326 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:32.801041) v2 ==== 47+0+0 (4092895305 0 0) 0x224a3660 con 0x1eb9d6a0 -3649> 2014-11-08 18:15:32.803851 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.801041) v2 -- ?+0 0x2127ecc0 con 0x1eb9c780 -3648> 2014-11-08 18:15:32.804153 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:32.801041) v2 -- ?+0 0x2234eae0 con 0x1eb9d6a0 -3647> 2014-11-08 18:15:33.050178 7fc5c362f700 5 osd.9 5773 tick -3646> 2014-11-08 18:15:33.050197 7fc5c362f700 -1 osd.9 5773 heartbeat_check: no reply from osd.4 since back 2014-11-08 18:15:04.751988 front 2014-11-08 18:15:04.751988 (cutoff 2014-11-08 18:15:13.050195) -3645> 2014-11-08 18:15:33.444966 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8628 ==== MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (1631924827 0 0) 0x1d3af400 con 0x1c5c0580 -3644> 2014-11-08 18:15:33.445016 7fc5a152e700 5 -- op tracker -- seq: 47198, time: 2014-11-08 18:15:33.226899, event: header_read, op: MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3643> 2014-11-08 18:15:33.445035 7fc5a152e700 5 -- op tracker -- seq: 47198, time: 2014-11-08 18:15:33.226904, event: throttled, op: MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3642> 2014-11-08 18:15:33.445049 7fc5a152e700 5 -- op tracker -- seq: 47198, time: 2014-11-08 18:15:33.444954, event: all_read, op: MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3641> 2014-11-08 18:15:33.445062 7fc5a152e700 5 -- op tracker -- seq: 47198, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3640> 2014-11-08 18:15:33.445109 7fc5afb15700 5 -- op tracker -- seq: 47198, time: 2014-11-08 18:15:33.445109, event: reached_pg, op: MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3639> 2014-11-08 18:15:33.445242 7fc5afb15700 5 -- op tracker -- seq: 47198, time: 2014-11-08 18:15:33.445242, event: done, op: MOSDPGPush(4.16s2 5773 [PushOp(a4ea9b16/1000001a021.00000bc6/head//4, version: 397'1618, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a4ea9b16/1000001a021.00000bc6/head//4@397'1618, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c8b9b16/1000002951f.00000009/head//4, version: 3033'4083, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c8b9b16/1000002951f.00000009/head//4@3033'4083, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(524c9b16/1000002a73c.000002c0/head//4, version: 3458'5326, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(524c9b16/1000002a73c.000002c0/head//4@3458'5326, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -3638> 2014-11-08 18:15:33.619647 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.4 192.168.1.16:0/31236 516 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:33.616935) v2 ==== 47+0+0 (1376864390 0 0) 0x2127d640 con 0x1eb9c8e0 -3637> 2014-11-08 18:15:33.619675 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:33.616935) v2 -- ?+0 0x224a5820 con 0x1eb9c8e0 -3636> 2014-11-08 18:15:33.620173 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.4 192.168.1.16:0/31236 516 ==== osd_ping(ping e5773 stamp 2014-11-08 18:15:33.616935) v2 ==== 47+0+0 (1376864390 0 0) 0x1e6ab840 con 0x1eb9d800 -3635> 2014-11-08 18:15:33.620189 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.16:0/31236 -- osd_ping(ping_reply e5773 stamp 2014-11-08 18:15:33.616935) v2 -- ?+0 0x224a3660 con 0x1eb9d800 -3634> 2014-11-08 18:15:33.884482 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6809/5031236 pipe(0x1cd62680 sd=69 :50049 s=2 pgs=8208 cs=1 l=0 c=0x21468c60).reader couldn't read tag, (0) Success -3633> 2014-11-08 18:15:33.884511 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6809/5031236 pipe(0x1cd62680 sd=69 :50049 s=2 pgs=8208 cs=1 l=0 c=0x21468c60).fault (0) Success -3632> 2014-11-08 18:15:33.884525 7fc5a39d2700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6809/5031236 pipe(0x1cd62680 sd=69 :50049 s=2 pgs=8208 cs=1 l=0 c=0x21468c60).fault with nothing to send, going to standby -3631> 2014-11-08 18:15:33.888030 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c03c0 con 0x1c5c0580 -3630> 2014-11-08 18:15:33.888049 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c3a20 con 0x1c5c0580 -3629> 2014-11-08 18:15:33.888056 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5773) v1 -- ?+0 0x1f4c23a0 con 0x1c5c0580 -3628> 2014-11-08 18:15:33.888064 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5773 [PushReplyOp(a4ea9b16/1000001a021.00000bc6/head//4),PushReplyOp(c8b9b16/1000002951f.00000009/head//4),PushReplyOp(524c9b16/1000002a73c.000002c0/head//4)]) v2 -- ?+0 0x1d3a9400 con 0x1c5c0580 -3627> 2014-11-08 18:15:33.892027 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8629 ==== pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) v3 ==== 757+0+0 (2590752163 0 0) 0x1d3ca880 con 0x1c5c0580 -3626> 2014-11-08 18:15:33.892097 7fc5a152e700 5 -- op tracker -- seq: 47199, time: 2014-11-08 18:15:33.891933, event: header_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3625> 2014-11-08 18:15:33.892107 7fc5a152e700 5 -- op tracker -- seq: 47199, time: 2014-11-08 18:15:33.891936, event: throttled, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3624> 2014-11-08 18:15:33.892110 7fc5a152e700 5 -- op tracker -- seq: 47199, time: 2014-11-08 18:15:33.892021, event: all_read, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3623> 2014-11-08 18:15:33.892113 7fc5a152e700 5 -- op tracker -- seq: 47199, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3622> 2014-11-08 18:15:33.892155 7fc5ad310700 5 -- op tracker -- seq: 47199, time: 2014-11-08 18:15:33.892155, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3621> 2014-11-08 18:15:33.892164 7fc5ad310700 5 -- op tracker -- seq: 47199, time: 2014-11-08 18:15:33.892164, event: started, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3620> 2014-11-08 18:15:33.892215 7fc5ad310700 5 -- op tracker -- seq: 47199, time: 2014-11-08 18:15:33.892215, event: done, op: pg_backfill(progress 4.16s2 e 5773/5773 lb 524c9b16/1000002a73c.000002c0/head//4) -3619> 2014-11-08 18:15:33.897604 7fc5a4ce5700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x1db0bc80 sd=63 :37524 s=2 pgs=48 cs=1 l=1 c=0x1cbf9ce0).reader couldn't read tag, (0) Success -3618> 2014-11-08 18:15:33.897656 7fc5a4ce5700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x1db0bc80 sd=63 :37524 s=2 pgs=48 cs=1 l=1 c=0x1cbf9ce0).fault (0) Success -3617> 2014-11-08 18:15:33.898039 7fc5a4be4700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x213c6300 sd=83 :45165 s=2 pgs=49 cs=1 l=1 c=0x1d603020).reader couldn't read tag, (0) Success -3616> 2014-11-08 18:15:33.898066 7fc5a4be4700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x213c6300 sd=83 :45165 s=2 pgs=49 cs=1 l=1 c=0x1d603020).fault (0) Success -3615> 2014-11-08 18:15:33.898542 7fc5a29c2700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/31236 pipe(0x1cd60000 sd=99 :6812 s=2 pgs=10649 cs=1 l=1 c=0x1eb9d800).reader couldn't read tag, (0) Success -3614> 2014-11-08 18:15:33.898559 7fc5a29c2700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/31236 pipe(0x1cd60000 sd=99 :6812 s=2 pgs=10649 cs=1 l=1 c=0x1eb9d800).fault (0) Success -3613> 2014-11-08 18:15:33.905100 7fc5a28c1700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/31236 pipe(0x1cd60840 sd=118 :6813 s=2 pgs=10650 cs=1 l=1 c=0x1eb9c8e0).reader couldn't read tag, (0) Success -3612> 2014-11-08 18:15:33.905153 7fc5a28c1700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/31236 pipe(0x1cd60840 sd=118 :6813 s=2 pgs=10650 cs=1 l=1 c=0x1eb9c8e0).fault (0) Success -3611> 2014-11-08 18:15:33.910624 7fc5a28c1700 1 -- 192.168.1.18:6812/4928 >> :/0 pipe(0x1d398b00 sd=63 :6812 s=0 pgs=0 cs=0 l=0 c=0x1eb9d540).accept sd=63 192.168.1.16:47540/0 -3610> 2014-11-08 18:15:33.910673 7fc5a6bfd700 1 -- 192.168.1.18:6813/4928 >> :/0 pipe(0x215a7bc0 sd=83 :6813 s=0 pgs=0 cs=0 l=0 c=0x1eb9c620).accept sd=83 192.168.1.16:54598/0 -3609> 2014-11-08 18:15:33.914107 7fc5a28c1700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/31236 pipe(0x1d398b00 sd=63 :6812 s=2 pgs=10664 cs=1 l=1 c=0x1eb9d540).reader couldn't read tag, (0) Success -3608> 2014-11-08 18:15:33.914160 7fc5a28c1700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/31236 pipe(0x1d398b00 sd=63 :6812 s=2 pgs=10664 cs=1 l=1 c=0x1eb9d540).fault (0) Success -3607> 2014-11-08 18:15:33.914173 7fc5a6bfd700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/31236 pipe(0x215a7bc0 sd=83 :6813 s=2 pgs=10665 cs=1 l=1 c=0x1eb9c620).reader couldn't read tag, (0) Success -3606> 2014-11-08 18:15:33.914207 7fc5a6bfd700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/31236 pipe(0x215a7bc0 sd=83 :6813 s=2 pgs=10665 cs=1 l=1 c=0x1eb9c620).fault (0) Success -3605> 2014-11-08 18:15:33.924855 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12070 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x228a0480 con 0x1c5c2680 -3604> 2014-11-08 18:15:34.033456 7fc5b8326700 1 -- 192.168.1.18:0/4928 mark_down 0x1d603020 -- pipe dne -3603> 2014-11-08 18:15:34.034126 7fc5a27c0700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x1d22b440 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1e50c200).connect error 192.168.1.16:6812/5031236, (111) Connection refused -3602> 2014-11-08 18:15:34.034184 7fc5a27c0700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x1d22b440 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1e50c200).fault (111) Connection refused -3601> 2014-11-08 18:15:34.034198 7fc5a27c0700 0 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x1d22b440 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1e50c200).fault -3600> 2014-11-08 18:15:34.034273 7fc5a28c1700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x23bd7900 sd=83 :0 s=1 pgs=0 cs=0 l=1 c=0x1cae0840).connect error 192.168.1.16:6811/5031236, (111) Connection refused -3599> 2014-11-08 18:15:34.034295 7fc5a28c1700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x23bd7900 sd=83 :0 s=1 pgs=0 cs=0 l=1 c=0x1cae0840).fault (111) Connection refused -3598> 2014-11-08 18:15:34.034307 7fc5a28c1700 0 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x23bd7900 sd=83 :0 s=1 pgs=0 cs=0 l=1 c=0x1cae0840).fault -3597> 2014-11-08 18:15:34.034544 7fc5a27c0700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x1d22b440 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1e50c200).connect error 192.168.1.16:6812/5031236, (111) Connection refused -3596> 2014-11-08 18:15:34.034573 7fc5a27c0700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6812/5031236 pipe(0x1d22b440 sd=63 :0 s=1 pgs=0 cs=0 l=1 c=0x1e50c200).fault (111) Connection refused -3595> 2014-11-08 18:15:34.034590 7fc5a28c1700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x23bd7900 sd=83 :0 s=1 pgs=0 cs=0 l=1 c=0x1cae0840).connect error 192.168.1.16:6811/5031236, (111) Connection refused -3594> 2014-11-08 18:15:34.034604 7fc5a28c1700 2 -- 192.168.1.18:0/4928 >> 192.168.1.16:6811/5031236 pipe(0x23bd7900 sd=83 :0 s=1 pgs=0 cs=0 l=1 c=0x1cae0840).fault (111) Connection refused -3593> 2014-11-08 18:15:34.040621 7fc5b9b29700 3 osd.9 5773 handle_osd_map epochs [5774,5774], i have 5773, src has [4761,5774] -3592> 2014-11-08 18:15:34.055109 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 192.168.1.16:6809/5031236 -- 0x1cd62680 -3591> 2014-11-08 18:15:34.055142 7fc5b9b29700 1 -- 192.168.1.18:0/4928 mark_down 0x1cae0840 -- 0x23bd7900 -3590> 2014-11-08 18:15:34.055151 7fc5b9b29700 1 -- 192.168.1.18:0/4928 mark_down 0x1e50c200 -- 0x1d22b440 -3589> 2014-11-08 18:15:34.091194 7fc5b9b29700 1 osd.9 5774 ms_handle_reset con 0x21468c60 session 0x211d6400 -3588> 2014-11-08 18:15:34.091237 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12071 ==== pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) v5 ==== 17844+0+0 (1000041686 0 0) 0x22e9da00 con 0x1c5c2680 -3587> 2014-11-08 18:15:34.091250 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:33.926177, event: header_read, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3586> 2014-11-08 18:15:34.091260 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:33.926178, event: throttled, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3585> 2014-11-08 18:15:34.091264 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:33.928989, event: all_read, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3584> 2014-11-08 18:15:34.091268 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:34.091248, event: dispatched, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3583> 2014-11-08 18:15:34.091272 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:34.091272, event: waiting_for_osdmap, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3582> 2014-11-08 18:15:34.091281 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:34.091281, event: started, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3581> 2014-11-08 18:15:34.091342 7fc5b9b29700 5 -- op tracker -- seq: 47200, time: 2014-11-08 18:15:34.091342, event: done, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -3580> 2014-11-08 18:15:34.091471 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8630 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x1ea7e300 con 0x1c5c0580 -3579> 2014-11-08 18:15:34.091484 7fc5b9b29700 3 osd.9 5774 handle_osd_map epochs [5774,5774], i have 5774, src has [4761,5774] -3578> 2014-11-08 18:15:34.091489 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12072 ==== pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) v3 ==== 400+0+0 (2451339434 0 0) 0x2485cec0 con 0x1c5c2680 -3577> 2014-11-08 18:15:34.091496 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:33.929008, event: header_read, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3576> 2014-11-08 18:15:34.091499 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:33.929009, event: throttled, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3575> 2014-11-08 18:15:34.091502 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:33.929092, event: all_read, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3574> 2014-11-08 18:15:34.091504 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:34.091495, event: dispatched, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3573> 2014-11-08 18:15:34.091507 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:34.091507, event: waiting_for_osdmap, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3572> 2014-11-08 18:15:34.091511 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:34.091511, event: started, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3571> 2014-11-08 18:15:34.091527 7fc5b9b29700 5 -- op tracker -- seq: 47201, time: 2014-11-08 18:15:34.091526, event: done, op: pg_query(4.ds1,4.12s1,4.24s1 epoch 5774) -3570> 2014-11-08 18:15:34.091534 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8631 ==== pg_notify(4.28s1(7) epoch 5774) v5 ==== 1168+0+0 (936742091 0 0) 0x1f4c23a0 con 0x1c5c0580 -3569> 2014-11-08 18:15:34.091540 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.060618, event: header_read, op: pg_notify(4.28s1(7) epoch 5774) -3568> 2014-11-08 18:15:34.091543 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.060619, event: throttled, op: pg_notify(4.28s1(7) epoch 5774) -3567> 2014-11-08 18:15:34.091545 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.060677, event: all_read, op: pg_notify(4.28s1(7) epoch 5774) -3566> 2014-11-08 18:15:34.091548 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.091539, event: dispatched, op: pg_notify(4.28s1(7) epoch 5774) -3565> 2014-11-08 18:15:34.091550 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.091550, event: waiting_for_osdmap, op: pg_notify(4.28s1(7) epoch 5774) -3564> 2014-11-08 18:15:34.091553 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.091553, event: started, op: pg_notify(4.28s1(7) epoch 5774) -3563> 2014-11-08 18:15:34.091564 7fc5b9b29700 5 -- op tracker -- seq: 47202, time: 2014-11-08 18:15:34.091564, event: done, op: pg_notify(4.28s1(7) epoch 5774) -3562> 2014-11-08 18:15:34.091572 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12073 ==== pg_notify(4.bfs0(41) epoch 5774) v5 ==== 3174+0+0 (3812159275 0 0) 0x2485d460 con 0x1c5c2680 -3561> 2014-11-08 18:15:34.091578 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:33.929107, event: header_read, op: pg_notify(4.bfs0(41) epoch 5774) -3560> 2014-11-08 18:15:34.091581 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:33.929108, event: throttled, op: pg_notify(4.bfs0(41) epoch 5774) -3559> 2014-11-08 18:15:34.091583 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:33.929159, event: all_read, op: pg_notify(4.bfs0(41) epoch 5774) -3558> 2014-11-08 18:15:34.091585 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:34.091577, event: dispatched, op: pg_notify(4.bfs0(41) epoch 5774) -3557> 2014-11-08 18:15:34.091588 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:34.091588, event: waiting_for_osdmap, op: pg_notify(4.bfs0(41) epoch 5774) -3556> 2014-11-08 18:15:34.091591 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:34.091591, event: started, op: pg_notify(4.bfs0(41) epoch 5774) -3555> 2014-11-08 18:15:34.091605 7fc5b9b29700 5 -- op tracker -- seq: 47203, time: 2014-11-08 18:15:34.091605, event: done, op: pg_notify(4.bfs0(41) epoch 5774) -3554> 2014-11-08 18:15:34.091619 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8632 ==== pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) v5 ==== 7689+0+0 (3953368822 0 0) 0x1f4c3a20 con 0x1c5c0580 -3553> 2014-11-08 18:15:34.091626 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.065642, event: header_read, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3552> 2014-11-08 18:15:34.091630 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.065643, event: throttled, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3551> 2014-11-08 18:15:34.091633 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.065774, event: all_read, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3550> 2014-11-08 18:15:34.091635 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.091626, event: dispatched, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3549> 2014-11-08 18:15:34.091639 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.091639, event: waiting_for_osdmap, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3548> 2014-11-08 18:15:34.091642 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.091642, event: started, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3547> 2014-11-08 18:15:34.091665 7fc5b9b29700 5 -- op tracker -- seq: 47204, time: 2014-11-08 18:15:34.091665, event: done, op: pg_notify(4.5s1(7),4.bfs2(41),4.b3s2(44) epoch 5774) -3546> 2014-11-08 18:15:34.091694 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12074 ==== pg_query(4.7s2,4.99s1 epoch 5774) v3 ==== 272+0+0 (3779568167 0 0) 0x2485a760 con 0x1c5c2680 -3545> 2014-11-08 18:15:34.091701 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:33.929166, event: header_read, op: pg_query(4.7s2,4.99s1 epoch 5774) -3544> 2014-11-08 18:15:34.091705 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:33.929167, event: throttled, op: pg_query(4.7s2,4.99s1 epoch 5774) -3543> 2014-11-08 18:15:34.091707 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:33.929189, event: all_read, op: pg_query(4.7s2,4.99s1 epoch 5774) -3542> 2014-11-08 18:15:34.091709 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:34.091700, event: dispatched, op: pg_query(4.7s2,4.99s1 epoch 5774) -3541> 2014-11-08 18:15:34.091712 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:34.091712, event: waiting_for_osdmap, op: pg_query(4.7s2,4.99s1 epoch 5774) -3540> 2014-11-08 18:15:34.091716 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:34.091716, event: started, op: pg_query(4.7s2,4.99s1 epoch 5774) -3539> 2014-11-08 18:15:34.091729 7fc5b9b29700 5 -- op tracker -- seq: 47205, time: 2014-11-08 18:15:34.091729, event: done, op: pg_query(4.7s2,4.99s1 epoch 5774) -3538> 2014-11-08 18:15:34.091735 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8633 ==== pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) v3 ==== 656+0+0 (963617993 0 0) 0x1f4c03c0 con 0x1c5c0580 -3537> 2014-11-08 18:15:34.091744 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.065783, event: header_read, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3536> 2014-11-08 18:15:34.091747 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.065784, event: throttled, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3535> 2014-11-08 18:15:34.091750 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.065810, event: all_read, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3534> 2014-11-08 18:15:34.091753 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.091743, event: dispatched, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3533> 2014-11-08 18:15:34.091757 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.091757, event: waiting_for_osdmap, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3532> 2014-11-08 18:15:34.091761 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.091761, event: started, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3531> 2014-11-08 18:15:34.091781 7fc5b9b29700 5 -- op tracker -- seq: 47206, time: 2014-11-08 18:15:34.091781, event: done, op: pg_query(4.6s2,4.b2s2,4.b5s1,4.bcs1,4.bds1 epoch 5774) -3530> 2014-11-08 18:15:34.091787 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12075 ==== pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) v5 ==== 3170+0+0 (409481662 0 0) 0x2485c560 con 0x1c5c2680 -3529> 2014-11-08 18:15:34.091794 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:33.933941, event: header_read, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3528> 2014-11-08 18:15:34.091797 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:33.933945, event: throttled, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3527> 2014-11-08 18:15:34.091800 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:33.934053, event: all_read, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3526> 2014-11-08 18:15:34.091802 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:34.091794, event: dispatched, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3525> 2014-11-08 18:15:34.091806 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:34.091805, event: waiting_for_osdmap, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3524> 2014-11-08 18:15:34.091809 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:34.091809, event: started, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3523> 2014-11-08 18:15:34.091826 7fc5b9b29700 5 -- op tracker -- seq: 47207, time: 2014-11-08 18:15:34.091826, event: done, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5774) -3522> 2014-11-08 18:15:34.091837 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8634 ==== pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) v5 ==== 8766+0+0 (3321689957 0 0) 0x1f4c2580 con 0x1c5c0580 -3521> 2014-11-08 18:15:34.091845 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.069771, event: header_read, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3520> 2014-11-08 18:15:34.091849 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.069772, event: throttled, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3519> 2014-11-08 18:15:34.091852 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.069974, event: all_read, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3518> 2014-11-08 18:15:34.091855 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.091845, event: dispatched, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3517> 2014-11-08 18:15:34.091859 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.091859, event: waiting_for_osdmap, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3516> 2014-11-08 18:15:34.091863 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.091863, event: started, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3515> 2014-11-08 18:15:34.091889 7fc5b9b29700 5 -- op tracker -- seq: 47208, time: 2014-11-08 18:15:34.091889, event: done, op: pg_notify(4.a0s2(21),4.98s2(24),4.9ds2(46),4.93s1(7) epoch 5774) -3514> 2014-11-08 18:15:34.091917 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12076 ==== pg_query(4.8ds1,4.9fs2 epoch 5774) v3 ==== 272+0+0 (1608350292 0 0) 0x2127eae0 con 0x1c5c2680 -3513> 2014-11-08 18:15:34.091924 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:33.934100, event: header_read, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3512> 2014-11-08 18:15:34.091927 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:33.934101, event: throttled, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3511> 2014-11-08 18:15:34.091930 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:33.934125, event: all_read, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3510> 2014-11-08 18:15:34.091932 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:34.091924, event: dispatched, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3509> 2014-11-08 18:15:34.091935 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:34.091935, event: waiting_for_osdmap, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3508> 2014-11-08 18:15:34.091938 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:34.091938, event: started, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3507> 2014-11-08 18:15:34.091948 7fc5b9b29700 5 -- op tracker -- seq: 47209, time: 2014-11-08 18:15:34.091948, event: done, op: pg_query(4.8ds1,4.9fs2 epoch 5774) -3506> 2014-11-08 18:15:34.091954 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8635 ==== pg_query(4.94s1,4.9cs2 epoch 5774) v3 ==== 272+0+0 (1260576628 0 0) 0x1f4c4b00 con 0x1c5c0580 -3505> 2014-11-08 18:15:34.091961 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.069982, event: header_read, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3504> 2014-11-08 18:15:34.091964 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.069983, event: throttled, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3503> 2014-11-08 18:15:34.091966 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.070006, event: all_read, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3502> 2014-11-08 18:15:34.091969 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.091960, event: dispatched, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3501> 2014-11-08 18:15:34.091971 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.091971, event: waiting_for_osdmap, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3500> 2014-11-08 18:15:34.091975 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.091975, event: started, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3499> 2014-11-08 18:15:34.091984 7fc5b9b29700 5 -- op tracker -- seq: 47210, time: 2014-11-08 18:15:34.091984, event: done, op: pg_query(4.94s1,4.9cs2 epoch 5774) -3498> 2014-11-08 18:15:34.091991 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12077 ==== pg_notify(4.75s1(7) epoch 5774) v5 ==== 1168+0+0 (3771053112 0 0) 0x2127e540 con 0x1c5c2680 -3497> 2014-11-08 18:15:34.091997 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:33.975604, event: header_read, op: pg_notify(4.75s1(7) epoch 5774) -3496> 2014-11-08 18:15:34.091999 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:33.975607, event: throttled, op: pg_notify(4.75s1(7) epoch 5774) -3495> 2014-11-08 18:15:34.092002 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:33.975694, event: all_read, op: pg_notify(4.75s1(7) epoch 5774) -3494> 2014-11-08 18:15:34.092004 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:34.091996, event: dispatched, op: pg_notify(4.75s1(7) epoch 5774) -3493> 2014-11-08 18:15:34.092006 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:34.092006, event: waiting_for_osdmap, op: pg_notify(4.75s1(7) epoch 5774) -3492> 2014-11-08 18:15:34.092009 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:34.092009, event: started, op: pg_notify(4.75s1(7) epoch 5774) -3491> 2014-11-08 18:15:34.092019 7fc5b9b29700 5 -- op tracker -- seq: 47211, time: 2014-11-08 18:15:34.092019, event: done, op: pg_notify(4.75s1(7) epoch 5774) -3490> 2014-11-08 18:15:34.092027 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8636 ==== pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) v3 ==== 400+0+0 (1186354968 0 0) 0x1f4c30c0 con 0x1c5c0580 -3489> 2014-11-08 18:15:34.092034 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.073388, event: header_read, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3488> 2014-11-08 18:15:34.092037 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.073389, event: throttled, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3487> 2014-11-08 18:15:34.092039 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.073442, event: all_read, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3486> 2014-11-08 18:15:34.092043 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.092033, event: dispatched, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3485> 2014-11-08 18:15:34.092046 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.092046, event: waiting_for_osdmap, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3484> 2014-11-08 18:15:34.092049 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.092049, event: started, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3483> 2014-11-08 18:15:34.092062 7fc5b9b29700 5 -- op tracker -- seq: 47212, time: 2014-11-08 18:15:34.092062, event: done, op: pg_query(4.61s2,4.7cs2,4.86s2 epoch 5774) -3482> 2014-11-08 18:15:34.092068 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12078 ==== pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) v3 ==== 400+0+0 (3149745947 0 0) 0x2127e900 con 0x1c5c2680 -3481> 2014-11-08 18:15:34.092075 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:33.977904, event: header_read, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3480> 2014-11-08 18:15:34.092078 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:33.977906, event: throttled, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3479> 2014-11-08 18:15:34.092081 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:33.977946, event: all_read, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3478> 2014-11-08 18:15:34.092084 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:34.092074, event: dispatched, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3477> 2014-11-08 18:15:34.092087 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:34.092087, event: waiting_for_osdmap, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3476> 2014-11-08 18:15:34.092090 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:34.092090, event: started, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3475> 2014-11-08 18:15:34.092103 7fc5b9b29700 5 -- op tracker -- seq: 47213, time: 2014-11-08 18:15:34.092103, event: done, op: pg_query(4.59s2,4.5as2,4.6ds2 epoch 5774) -3474> 2014-11-08 18:15:34.092109 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8637 ==== pg_notify(4.21s1(50),4.as1(7) epoch 5774) v5 ==== 4861+0+0 (3947505017 0 0) 0x1f4c6180 con 0x1c5c0580 -3473> 2014-11-08 18:15:34.092116 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.073763, event: header_read, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3472> 2014-11-08 18:15:34.092119 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.073764, event: throttled, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3471> 2014-11-08 18:15:34.092121 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.073848, event: all_read, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3470> 2014-11-08 18:15:34.092124 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.092115, event: dispatched, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3469> 2014-11-08 18:15:34.092127 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.092127, event: waiting_for_osdmap, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3468> 2014-11-08 18:15:34.092130 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.092130, event: started, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3467> 2014-11-08 18:15:34.092146 7fc5b9b29700 5 -- op tracker -- seq: 47214, time: 2014-11-08 18:15:34.092146, event: done, op: pg_notify(4.21s1(50),4.as1(7) epoch 5774) -3466> 2014-11-08 18:15:34.092161 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12079 ==== MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 v3 ==== 30+0+0 (2361839719 0 0) 0x21278000 con 0x1c5c2680 -3465> 2014-11-08 18:15:34.092167 7fc5b9b29700 5 -- op tracker -- seq: 47215, time: 2014-11-08 18:15:33.978558, event: header_read, op: MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 -3464> 2014-11-08 18:15:34.092170 7fc5b9b29700 5 -- op tracker -- seq: 47215, time: 2014-11-08 18:15:33.978559, event: throttled, op: MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 -3463> 2014-11-08 18:15:34.092172 7fc5b9b29700 5 -- op tracker -- seq: 47215, time: 2014-11-08 18:15:33.978584, event: all_read, op: MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 -3462> 2014-11-08 18:15:34.092174 7fc5b9b29700 5 -- op tracker -- seq: 47215, time: 2014-11-08 18:15:34.092167, event: dispatched, op: MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 -3461> 2014-11-08 18:15:34.092177 7fc5b9b29700 5 -- op tracker -- seq: 47215, time: 2014-11-08 18:15:34.092177, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 -3460> 2014-11-08 18:15:34.092189 7fc5b9b29700 5 -- op tracker -- seq: 47215, time: 2014-11-08 18:15:34.092189, event: done, op: MBackfillReserve GRANT pgid: 4.20s0, query_epoch: 5774 -3459> 2014-11-08 18:15:34.092196 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8638 ==== pg_query(4.14s2,4.2ds2 epoch 5774) v3 ==== 272+0+0 (3335679248 0 0) 0x1f4c2b20 con 0x1c5c0580 -3458> 2014-11-08 18:15:34.092202 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.073907, event: header_read, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3457> 2014-11-08 18:15:34.092205 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.073907, event: throttled, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3456> 2014-11-08 18:15:34.092207 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.073931, event: all_read, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3455> 2014-11-08 18:15:34.092209 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.092201, event: dispatched, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3454> 2014-11-08 18:15:34.092212 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.092212, event: waiting_for_osdmap, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3453> 2014-11-08 18:15:34.092216 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.092215, event: started, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3452> 2014-11-08 18:15:34.092226 7fc5b9b29700 5 -- op tracker -- seq: 47216, time: 2014-11-08 18:15:34.092226, event: done, op: pg_query(4.14s2,4.2ds2 epoch 5774) -3451> 2014-11-08 18:15:34.092232 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12080 ==== MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 v3 ==== 30+0+0 (3241948086 0 0) 0x2127f440 con 0x1c5c2680 -3450> 2014-11-08 18:15:34.092238 7fc5b9b29700 5 -- op tracker -- seq: 47217, time: 2014-11-08 18:15:33.978592, event: header_read, op: MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 -3449> 2014-11-08 18:15:34.092241 7fc5b9b29700 5 -- op tracker -- seq: 47217, time: 2014-11-08 18:15:33.978593, event: throttled, op: MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 -3448> 2014-11-08 18:15:34.092243 7fc5b9b29700 5 -- op tracker -- seq: 47217, time: 2014-11-08 18:15:33.978611, event: all_read, op: MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 -3447> 2014-11-08 18:15:34.092245 7fc5b9b29700 5 -- op tracker -- seq: 47217, time: 2014-11-08 18:15:34.092237, event: dispatched, op: MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 -3446> 2014-11-08 18:15:34.092248 7fc5b9b29700 5 -- op tracker -- seq: 47217, time: 2014-11-08 18:15:34.092248, event: waiting_for_osdmap, op: MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 -3445> 2014-11-08 18:15:34.092255 7fc5b9b29700 5 -- op tracker -- seq: 47217, time: 2014-11-08 18:15:34.092255, event: done, op: MBackfillReserve REQUEST pgid: 4.88s1, query_epoch: 5774, prio: 201 -3444> 2014-11-08 18:15:34.092261 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8639 ==== MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 v3 ==== 30+0+0 (3748052399 0 0) 0x1f4c2ee0 con 0x1c5c0580 -3443> 2014-11-08 18:15:34.092267 7fc5b9b29700 5 -- op tracker -- seq: 47218, time: 2014-11-08 18:15:34.077400, event: header_read, op: MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 -3442> 2014-11-08 18:15:34.092269 7fc5b9b29700 5 -- op tracker -- seq: 47218, time: 2014-11-08 18:15:34.077402, event: throttled, op: MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 -3441> 2014-11-08 18:15:34.092272 7fc5b9b29700 5 -- op tracker -- seq: 47218, time: 2014-11-08 18:15:34.077453, event: all_read, op: MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 -3440> 2014-11-08 18:15:34.092274 7fc5b9b29700 5 -- op tracker -- seq: 47218, time: 2014-11-08 18:15:34.092266, event: dispatched, op: MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 -3439> 2014-11-08 18:15:34.092277 7fc5b9b29700 5 -- op tracker -- seq: 47218, time: 2014-11-08 18:15:34.092277, event: waiting_for_osdmap, op: MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 -3438> 2014-11-08 18:15:34.092283 7fc5b9b29700 5 -- op tracker -- seq: 47218, time: 2014-11-08 18:15:34.092283, event: done, op: MBackfillReserve REQUEST pgid: 4.89s2, query_epoch: 5774, prio: 201 -3437> 2014-11-08 18:15:34.092289 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12081 ==== MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 v3 ==== 30+0+0 (1871153083 0 0) 0x22a22ee0 con 0x1c5c2680 -3436> 2014-11-08 18:15:34.092295 7fc5b9b29700 5 -- op tracker -- seq: 47219, time: 2014-11-08 18:15:33.978616, event: header_read, op: MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 -3435> 2014-11-08 18:15:34.092298 7fc5b9b29700 5 -- op tracker -- seq: 47219, time: 2014-11-08 18:15:33.978616, event: throttled, op: MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 -3434> 2014-11-08 18:15:34.092300 7fc5b9b29700 5 -- op tracker -- seq: 47219, time: 2014-11-08 18:15:33.978634, event: all_read, op: MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 -3433> 2014-11-08 18:15:34.092302 7fc5b9b29700 5 -- op tracker -- seq: 47219, time: 2014-11-08 18:15:34.092295, event: dispatched, op: MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 -3432> 2014-11-08 18:15:34.092305 7fc5b9b29700 5 -- op tracker -- seq: 47219, time: 2014-11-08 18:15:34.092305, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 -3431> 2014-11-08 18:15:34.092313 7fc5b9b29700 5 -- op tracker -- seq: 47219, time: 2014-11-08 18:15:34.092313, event: done, op: MBackfillReserve GRANT pgid: 4.7bs0, query_epoch: 5774 -3430> 2014-11-08 18:15:34.092319 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8640 ==== pg_notify(4.36s0(40) epoch 5774) v5 ==== 3115+0+0 (1978206008 0 0) 0x2286ddc0 con 0x1c5c0580 -3429> 2014-11-08 18:15:34.092325 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.077798, event: header_read, op: pg_notify(4.36s0(40) epoch 5774) -3428> 2014-11-08 18:15:34.092327 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.077799, event: throttled, op: pg_notify(4.36s0(40) epoch 5774) -3427> 2014-11-08 18:15:34.092330 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.077849, event: all_read, op: pg_notify(4.36s0(40) epoch 5774) -3426> 2014-11-08 18:15:34.092332 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.092324, event: dispatched, op: pg_notify(4.36s0(40) epoch 5774) -3425> 2014-11-08 18:15:34.092334 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.092334, event: waiting_for_osdmap, op: pg_notify(4.36s0(40) epoch 5774) -3424> 2014-11-08 18:15:34.092337 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.092337, event: started, op: pg_notify(4.36s0(40) epoch 5774) -3423> 2014-11-08 18:15:34.092347 7fc5b9b29700 5 -- op tracker -- seq: 47220, time: 2014-11-08 18:15:34.092347, event: done, op: pg_notify(4.36s0(40) epoch 5774) -3422> 2014-11-08 18:15:34.092358 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12082 ==== pg_notify(4.36s0(40) epoch 5774) v5 ==== 3115+0+0 (460362246 0 0) 0x1e6ac560 con 0x1c5c2680 -3421> 2014-11-08 18:15:34.092364 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:33.978638, event: header_read, op: pg_notify(4.36s0(40) epoch 5774) -3420> 2014-11-08 18:15:34.092367 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:33.978638, event: throttled, op: pg_notify(4.36s0(40) epoch 5774) -3419> 2014-11-08 18:15:34.092369 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:33.978713, event: all_read, op: pg_notify(4.36s0(40) epoch 5774) -3418> 2014-11-08 18:15:34.092372 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:34.092363, event: dispatched, op: pg_notify(4.36s0(40) epoch 5774) -3417> 2014-11-08 18:15:34.092374 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:34.092374, event: waiting_for_osdmap, op: pg_notify(4.36s0(40) epoch 5774) -3416> 2014-11-08 18:15:34.092377 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:34.092377, event: started, op: pg_notify(4.36s0(40) epoch 5774) -3415> 2014-11-08 18:15:34.092386 7fc5b9b29700 5 -- op tracker -- seq: 47221, time: 2014-11-08 18:15:34.092386, event: done, op: pg_notify(4.36s0(40) epoch 5774) -3414> 2014-11-08 18:15:34.092398 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8641 ==== pg_query(4.2as2,4.33s2 epoch 5774) v3 ==== 272+0+0 (971156350 0 0) 0x2286e540 con 0x1c5c0580 -3413> 2014-11-08 18:15:34.092404 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.078081, event: header_read, op: pg_query(4.2as2,4.33s2 epoch 5774) -3412> 2014-11-08 18:15:34.092407 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.078081, event: throttled, op: pg_query(4.2as2,4.33s2 epoch 5774) -3411> 2014-11-08 18:15:34.092409 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.078104, event: all_read, op: pg_query(4.2as2,4.33s2 epoch 5774) -3410> 2014-11-08 18:15:34.092412 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.092403, event: dispatched, op: pg_query(4.2as2,4.33s2 epoch 5774) -3409> 2014-11-08 18:15:34.092414 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.092414, event: waiting_for_osdmap, op: pg_query(4.2as2,4.33s2 epoch 5774) -3408> 2014-11-08 18:15:34.092418 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.092418, event: started, op: pg_query(4.2as2,4.33s2 epoch 5774) -3407> 2014-11-08 18:15:34.092431 7fc5b9b29700 5 -- op tracker -- seq: 47222, time: 2014-11-08 18:15:34.092431, event: done, op: pg_query(4.2as2,4.33s2 epoch 5774) -3406> 2014-11-08 18:15:34.092437 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12083 ==== pg_query(4.67s1,4.67s2 epoch 5774) v3 ==== 272+0+0 (2922917095 0 0) 0x1e6ac740 con 0x1c5c2680 -3405> 2014-11-08 18:15:34.092444 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:33.980342, event: header_read, op: pg_query(4.67s1,4.67s2 epoch 5774) -3404> 2014-11-08 18:15:34.092446 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:33.980343, event: throttled, op: pg_query(4.67s1,4.67s2 epoch 5774) -3403> 2014-11-08 18:15:34.092449 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:33.980385, event: all_read, op: pg_query(4.67s1,4.67s2 epoch 5774) -3402> 2014-11-08 18:15:34.092451 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:34.092443, event: dispatched, op: pg_query(4.67s1,4.67s2 epoch 5774) -3401> 2014-11-08 18:15:34.092454 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:34.092454, event: waiting_for_osdmap, op: pg_query(4.67s1,4.67s2 epoch 5774) -3400> 2014-11-08 18:15:34.092458 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:34.092458, event: started, op: pg_query(4.67s1,4.67s2 epoch 5774) -3399> 2014-11-08 18:15:34.092467 7fc5b9b29700 5 -- op tracker -- seq: 47223, time: 2014-11-08 18:15:34.092467, event: done, op: pg_query(4.67s1,4.67s2 epoch 5774) -3398> 2014-11-08 18:15:34.092473 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8642 ==== pg_notify(4.65s2(68),4.38s1(53) epoch 5774) v5 ==== 8657+0+0 (2944206099 0 0) 0x2286d640 con 0x1c5c0580 -3397> 2014-11-08 18:15:34.092480 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.082133, event: header_read, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3396> 2014-11-08 18:15:34.092483 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.082134, event: throttled, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3395> 2014-11-08 18:15:34.092486 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.082325, event: all_read, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3394> 2014-11-08 18:15:34.092488 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.092479, event: dispatched, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3393> 2014-11-08 18:15:34.092491 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.092491, event: waiting_for_osdmap, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3392> 2014-11-08 18:15:34.092496 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.092496, event: started, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3391> 2014-11-08 18:15:34.092512 7fc5b9b29700 5 -- op tracker -- seq: 47224, time: 2014-11-08 18:15:34.092512, event: done, op: pg_notify(4.65s2(68),4.38s1(53) epoch 5774) -3390> 2014-11-08 18:15:34.092552 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12084 ==== pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) v5 ==== 10945+0+0 (1912930379 0 0) 0x1e6acce0 con 0x1c5c2680 -3389> 2014-11-08 18:15:34.092561 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.032232, event: header_read, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3388> 2014-11-08 18:15:34.092565 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.032234, event: throttled, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3387> 2014-11-08 18:15:34.092569 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.032403, event: all_read, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3386> 2014-11-08 18:15:34.092572 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.092560, event: dispatched, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3385> 2014-11-08 18:15:34.092584 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.092584, event: waiting_for_osdmap, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3384> 2014-11-08 18:15:34.092589 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.092589, event: started, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3383> 2014-11-08 18:15:34.092621 7fc5b9b29700 5 -- op tracker -- seq: 47225, time: 2014-11-08 18:15:34.092621, event: done, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7),4.31s2(22) epoch 5774) -3382> 2014-11-08 18:15:34.092655 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8643 ==== pg_query(4.49s2,4.5bs1 epoch 5774) v3 ==== 272+0+0 (903742319 0 0) 0x22868d20 con 0x1c5c0580 -3381> 2014-11-08 18:15:34.092662 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.082334, event: header_read, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3380> 2014-11-08 18:15:34.092665 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.082334, event: throttled, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3379> 2014-11-08 18:15:34.092668 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.082356, event: all_read, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3378> 2014-11-08 18:15:34.092670 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.092661, event: dispatched, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3377> 2014-11-08 18:15:34.092673 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.092673, event: waiting_for_osdmap, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3376> 2014-11-08 18:15:34.092676 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.092676, event: started, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3375> 2014-11-08 18:15:34.092888 7fc5b9b29700 5 -- op tracker -- seq: 47226, time: 2014-11-08 18:15:34.092887, event: done, op: pg_query(4.49s2,4.5bs1 epoch 5774) -3374> 2014-11-08 18:15:34.092902 7fc5c362f700 5 osd.9 5774 tick -3373> 2014-11-08 18:15:34.092922 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12085 ==== pg_query(4.45s2,4.57s1 epoch 5774) v3 ==== 272+0+0 (2223561424 0 0) 0x1e6ad0a0 con 0x1c5c2680 -3372> 2014-11-08 18:15:34.092932 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.032419, event: header_read, op: pg_query(4.45s2,4.57s1 epoch 5774) -3371> 2014-11-08 18:15:34.092937 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.032419, event: throttled, op: pg_query(4.45s2,4.57s1 epoch 5774) -3370> 2014-11-08 18:15:34.092941 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.032454, event: all_read, op: pg_query(4.45s2,4.57s1 epoch 5774) -3369> 2014-11-08 18:15:34.092945 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.092931, event: dispatched, op: pg_query(4.45s2,4.57s1 epoch 5774) -3368> 2014-11-08 18:15:34.092950 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.092950, event: waiting_for_osdmap, op: pg_query(4.45s2,4.57s1 epoch 5774) -3367> 2014-11-08 18:15:34.092955 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.092955, event: started, op: pg_query(4.45s2,4.57s1 epoch 5774) -3366> 2014-11-08 18:15:34.092968 7fc5b9b29700 5 -- op tracker -- seq: 47227, time: 2014-11-08 18:15:34.092968, event: done, op: pg_query(4.45s2,4.57s1 epoch 5774) -3365> 2014-11-08 18:15:34.111743 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1507 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x22ac9200 con 0x44ed6a0 -3364> 2014-11-08 18:15:34.111773 7fc5bb32c700 3 osd.9 5774 handle_osd_map epochs [5774,5774], i have 5774, src has [4761,5774] -3363> 2014-11-08 18:15:34.111782 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1508 ==== osd_map(5773..5774 src has 4761..5774) v3 ==== 393+0+0 (523978926 0 0) 0x22ac8fc0 con 0x44ed6a0 -3362> 2014-11-08 18:15:34.111790 7fc5bb32c700 3 osd.9 5774 handle_osd_map epochs [5773,5774], i have 5774, src has [4761,5774] -3361> 2014-11-08 18:15:34.113389 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1509 ==== osd_map(5770..5774 src has 4761..5774) v3 ==== 981+0+0 (3930518295 0 0) 0x22ac8d80 con 0x44ed6a0 -3360> 2014-11-08 18:15:34.113404 7fc5bb32c700 3 osd.9 5774 handle_osd_map epochs [5770,5774], i have 5774, src has [4761,5774] -3359> 2014-11-08 18:15:34.146057 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8320 lcod 5734'8326 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1661.559626 62 0.000628 -3358> 2014-11-08 18:15:34.146103 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8320 lcod 5734'8326 mlcod 0'0 active] exit Started/Primary/Active 1665.848093 0 0.000000 -3357> 2014-11-08 18:15:34.146120 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8320 lcod 5734'8326 mlcod 0'0 active] exit Started/Primary 1666.254533 0 0.000000 -3356> 2014-11-08 18:15:34.146165 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=4863-5750/34 luod=0'0 crt=5734'8305 active] exit Started/ReplicaActive/RepNotRecovering 1665.824857 65 0.000580 -3355> 2014-11-08 18:15:34.147375 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8320 lcod 5734'8326 mlcod 0'0 active] exit Started 1666.256847 0 0.000000 -3354> 2014-11-08 18:15:34.147399 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8320 lcod 5734'8326 mlcod 0'0 active] enter Reset -3353> 2014-11-08 18:15:34.147957 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=4863-5750/34 luod=0'0 crt=5734'8305 active] exit Started/ReplicaActive 1665.826659 0 0.000000 -3352> 2014-11-08 18:15:34.147973 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=4863-5750/34 luod=0'0 crt=5734'8305 active] exit Started 1666.285264 0 0.000000 -3351> 2014-11-08 18:15:34.147981 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=4863-5750/34 luod=0'0 crt=5734'8305 active] enter Reset -3350> 2014-11-08 18:15:34.150695 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=4863-5773/35 crt=5734'8305 inactive NOTIFY] exit Reset 0.002702 1 0.057895 -3349> 2014-11-08 18:15:34.150713 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=4863-5773/35 crt=5734'8305 inactive NOTIFY] enter Started -3348> 2014-11-08 18:15:34.150725 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=4863-5773/35 crt=5734'8305 inactive NOTIFY] enter Start -3347> 2014-11-08 18:15:34.150730 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=4863-5773/35 crt=5734'8305 inactive NOTIFY] state<Start>: transitioning to Stray -3346> 2014-11-08 18:15:34.150738 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=4863-5773/35 crt=5734'8305 inactive NOTIFY] exit Start 0.000013 0 0.000000 -3345> 2014-11-08 18:15:34.150760 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2as2( v 5734'8319 (3458'5318,5734'8319] local-les=5752 n=7705 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=4863-5773/35 crt=5734'8305 inactive NOTIFY] enter Started/Stray -3344> 2014-11-08 18:15:34.150886 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8245 lcod 5734'8246 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1664.514907 62 0.000319 -3343> 2014-11-08 18:15:34.150900 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8245 lcod 5734'8246 mlcod 0'0 active] exit Started/Primary/Active 1665.853723 0 0.000000 -3342> 2014-11-08 18:15:34.150909 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8245 lcod 5734'8246 mlcod 0'0 active] exit Started/Primary 1666.287807 0 0.000000 -3341> 2014-11-08 18:15:34.150916 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8245 lcod 5734'8246 mlcod 0'0 active] exit Started 1666.287898 0 0.000000 -3340> 2014-11-08 18:15:34.150923 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8245 lcod 5734'8246 mlcod 0'0 active] enter Reset -3339> 2014-11-08 18:15:34.150957 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] exit Reset 0.000034 1 0.000070 -3338> 2014-11-08 18:15:34.150974 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] enter Started -3337> 2014-11-08 18:15:34.150980 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] enter Start -3336> 2014-11-08 18:15:34.150985 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3335> 2014-11-08 18:15:34.150993 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] exit Start 0.000012 0 0.000000 -3334> 2014-11-08 18:15:34.151000 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] enter Started/Primary -3333> 2014-11-08 18:15:34.151007 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] enter Started/Primary/Peering -3332> 2014-11-08 18:15:34.151031 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] exit Reset 0.003632 1 0.058270 -3331> 2014-11-08 18:15:34.151043 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] enter Started -3330> 2014-11-08 18:15:34.151050 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] enter Start -3329> 2014-11-08 18:15:34.151056 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3328> 2014-11-08 18:15:34.151063 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] exit Start 0.000012 0 0.000000 -3327> 2014-11-08 18:15:34.151082 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] enter Started/Primary -3326> 2014-11-08 18:15:34.151088 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] enter Started/Primary/Peering -3325> 2014-11-08 18:15:34.165068 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3324> 2014-11-08 18:15:34.165225 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1651.173301 52 0.000192 -3323> 2014-11-08 18:15:34.165239 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active] exit Started/Primary/Active 1665.282575 0 0.000000 -3322> 2014-11-08 18:15:34.165251 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active] exit Started/Primary 1666.301753 0 0.000000 -3321> 2014-11-08 18:15:34.165259 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active] exit Started 1666.301787 0 0.000000 -3320> 2014-11-08 18:15:34.165266 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active] enter Reset -3319> 2014-11-08 18:15:34.165321 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] exit Reset 0.000055 1 0.000097 -3318> 2014-11-08 18:15:34.165331 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] enter Started -3317> 2014-11-08 18:15:34.165337 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] enter Start -3316> 2014-11-08 18:15:34.165343 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3315> 2014-11-08 18:15:34.165351 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] exit Start 0.000014 0 0.000000 -3314> 2014-11-08 18:15:34.165359 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] enter Started/Primary -3313> 2014-11-08 18:15:34.165366 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] enter Started/Primary/Peering -3312> 2014-11-08 18:15:34.165373 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3311> 2014-11-08 18:15:34.165424 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1651.125166 52 0.000196 -3310> 2014-11-08 18:15:34.165435 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active] exit Started/Primary/Active 1665.275513 0 0.000000 -3309> 2014-11-08 18:15:34.165443 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active] exit Started/Primary 1666.282293 0 0.000000 -3308> 2014-11-08 18:15:34.165450 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active] exit Started 1666.282327 0 0.000000 -3307> 2014-11-08 18:15:34.165457 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active] enter Reset -3306> 2014-11-08 18:15:34.165488 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] exit Reset 0.000031 1 0.000062 -3305> 2014-11-08 18:15:34.165497 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] enter Started -3304> 2014-11-08 18:15:34.165504 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] enter Start -3303> 2014-11-08 18:15:34.165510 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3302> 2014-11-08 18:15:34.165517 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] exit Start 0.000013 0 0.000000 -3301> 2014-11-08 18:15:34.165524 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] enter Started/Primary -3300> 2014-11-08 18:15:34.165531 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] enter Started/Primary/Peering -3299> 2014-11-08 18:15:34.165537 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3298> 2014-11-08 18:15:34.165597 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8164 active] exit Started/ReplicaActive/RepNotRecovering 1661.149324 58 0.000322 -3297> 2014-11-08 18:15:34.165608 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8164 active] exit Started/ReplicaActive 1661.149350 0 0.000000 -3296> 2014-11-08 18:15:34.165617 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8164 active] exit Started 1666.282311 0 0.000000 -3295> 2014-11-08 18:15:34.165636 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8164 active] enter Reset -3294> 2014-11-08 18:15:34.165671 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] exit Reset 0.000035 1 0.000062 -3293> 2014-11-08 18:15:34.165679 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] enter Started -3292> 2014-11-08 18:15:34.165696 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] enter Start -3291> 2014-11-08 18:15:34.165702 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] state<Start>: transitioning to Stray -3290> 2014-11-08 18:15:34.165709 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] exit Start 0.000013 0 0.000000 -3289> 2014-11-08 18:15:34.165717 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] enter Started/Stray -3288> 2014-11-08 18:15:34.165767 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=3 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded+remapped+backfilling] exit Started/Primary/Active/Backfilling 1025.960699 49 0.000192 -3287> 2014-11-08 18:15:34.165779 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=3 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active 1659.770984 0 0.000000 -3286> 2014-11-08 18:15:34.165795 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=3 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+remapped] exit Started/Primary 1661.166278 0 0.000000 -3285> 2014-11-08 18:15:34.165805 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=3 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+remapped] exit Started 1661.166318 0 0.000000 -3284> 2014-11-08 18:15:34.165812 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5751/5753/5728) [9,1,4]/[9,1,2147483647] r=0 lpr=5753 pi=5524-5752/13 rops=3 bft=4(2) crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+remapped] enter Reset -3283> 2014-11-08 18:15:34.165905 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] exit Reset 0.000093 1 0.000136 -3282> 2014-11-08 18:15:34.165915 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] enter Started -3281> 2014-11-08 18:15:34.165921 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] enter Start -3280> 2014-11-08 18:15:34.165927 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3279> 2014-11-08 18:15:34.165934 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] exit Start 0.000013 0 0.000000 -3278> 2014-11-08 18:15:34.165942 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] enter Started/Primary -3277> 2014-11-08 18:15:34.165948 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] enter Started/Primary/Peering -3276> 2014-11-08 18:15:34.165954 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3275> 2014-11-08 18:15:34.166008 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8162 active] exit Started/ReplicaActive/RepNotRecovering 1665.278928 65 0.000328 -3274> 2014-11-08 18:15:34.166018 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8162 active] exit Started/ReplicaActive 1665.278949 0 0.000000 -3273> 2014-11-08 18:15:34.166026 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8162 active] exit Started 1666.278384 0 0.000000 -3272> 2014-11-08 18:15:34.166033 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8162 active] enter Reset -3271> 2014-11-08 18:15:34.166128 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] exit Reset 0.000095 1 0.000048 -3270> 2014-11-08 18:15:34.166136 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] enter Started -3269> 2014-11-08 18:15:34.166142 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] enter Start -3268> 2014-11-08 18:15:34.166148 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] state<Start>: transitioning to Stray -3267> 2014-11-08 18:15:34.166155 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] exit Start 0.000012 0 0.000000 -3266> 2014-11-08 18:15:34.166161 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] enter Started/Stray -3265> 2014-11-08 18:15:34.166223 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8293 lcod 5734'8298 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1661.579287 62 0.000172 -3264> 2014-11-08 18:15:34.166233 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8293 lcod 5734'8298 mlcod 0'0 active] exit Started/Primary/Active 1665.867771 0 0.000000 -3263> 2014-11-08 18:15:34.166241 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8293 lcod 5734'8298 mlcod 0'0 active] exit Started/Primary 1666.278386 0 0.000000 -3262> 2014-11-08 18:15:34.166247 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8293 lcod 5734'8298 mlcod 0'0 active] exit Started 1666.278422 0 0.000000 -3261> 2014-11-08 18:15:34.166254 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5734'8293 lcod 5734'8298 mlcod 0'0 active] enter Reset -3260> 2014-11-08 18:15:34.166289 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] exit Reset 0.000035 1 0.000066 -3259> 2014-11-08 18:15:34.166298 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] enter Started -3258> 2014-11-08 18:15:34.166304 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] enter Start -3257> 2014-11-08 18:15:34.166309 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3256> 2014-11-08 18:15:34.166316 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] exit Start 0.000012 0 0.000000 -3255> 2014-11-08 18:15:34.166324 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] enter Started/Primary -3254> 2014-11-08 18:15:34.166330 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] enter Started/Primary/Peering -3253> 2014-11-08 18:15:34.166336 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3252> 2014-11-08 18:15:34.166380 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8268 active] exit Started/ReplicaActive/RepNotRecovering 1665.844492 65 0.000345 -3251> 2014-11-08 18:15:34.166390 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8268 active] exit Started/ReplicaActive 1665.844511 0 0.000000 -3250> 2014-11-08 18:15:34.166397 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8268 active] exit Started 1666.278393 0 0.000000 -3249> 2014-11-08 18:15:34.166404 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5432-5750/23 luod=0'0 crt=5734'8268 active] enter Reset -3248> 2014-11-08 18:15:34.166439 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8268 inactive NOTIFY] exit Reset 0.000035 1 0.000046 -3247> 2014-11-08 18:15:34.166447 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8268 inactive NOTIFY] enter Started -3246> 2014-11-08 18:15:34.166453 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8268 inactive NOTIFY] enter Start -3245> 2014-11-08 18:15:34.166458 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8268 inactive NOTIFY] state<Start>: transitioning to Stray -3244> 2014-11-08 18:15:34.166465 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8268 inactive NOTIFY] exit Start 0.000012 0 0.000000 -3243> 2014-11-08 18:15:34.166471 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6s2( v 5734'8276 (3458'5275,5734'8276] local-les=5752 n=7754 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8268 inactive NOTIFY] enter Started/Stray -3242> 2014-11-08 18:15:34.166544 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5751/5759/5728) [4,9,2]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/40 bft=2(2),4(0) crt=5734'8206 lcod 5734'8211 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 961.068191 33 0.000122 -3241> 2014-11-08 18:15:34.166557 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5751/5759/5728) [4,9,2]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/40 bft=2(2),4(0) crt=5734'8206 lcod 5734'8211 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 965.448028 0 0.000000 -3240> 2014-11-08 18:15:34.166568 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5751/5759/5728) [4,9,2]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/40 bft=2(2),4(0) crt=5734'8206 lcod 5734'8211 mlcod 0'0 active+remapped] exit Started/Primary 1015.776724 0 0.000000 -3239> 2014-11-08 18:15:34.166576 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5751/5759/5728) [4,9,2]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/40 bft=2(2),4(0) crt=5734'8206 lcod 5734'8211 mlcod 0'0 active+remapped] exit Started 1015.776780 0 0.000000 -3238> 2014-11-08 18:15:34.166585 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5751/5759/5728) [4,9,2]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/40 bft=2(2),4(0) crt=5734'8206 lcod 5734'8211 mlcod 0'0 active+remapped] enter Reset -3237> 2014-11-08 18:15:34.166620 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] exit Reset 0.000035 1 0.000075 -3236> 2014-11-08 18:15:34.166630 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] enter Started -3235> 2014-11-08 18:15:34.166636 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] enter Start -3234> 2014-11-08 18:15:34.166643 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] state<Start>: transitioning to Primary -3233> 2014-11-08 18:15:34.166651 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -3232> 2014-11-08 18:15:34.166658 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] enter Started/Primary -3231> 2014-11-08 18:15:34.166665 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] enter Started/Primary/Peering -3230> 2014-11-08 18:15:34.166672 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -3229> 2014-11-08 18:15:34.166946 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3228> 2014-11-08 18:15:34.167010 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/45 luod=0'0 crt=5734'8252 active] exit Started/ReplicaActive/RepNotRecovering 1665.859994 65 0.000277 -3227> 2014-11-08 18:15:34.167021 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/45 luod=0'0 crt=5734'8252 active] exit Started/ReplicaActive 1665.860014 0 0.000000 -3226> 2014-11-08 18:15:34.167029 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/45 luod=0'0 crt=5734'8252 active] exit Started 1666.303711 0 0.000000 -3225> 2014-11-08 18:15:34.167036 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/45 luod=0'0 crt=5734'8252 active] enter Reset -3224> 2014-11-08 18:15:34.167111 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/46 crt=5734'8252 inactive NOTIFY] exit Reset 0.000074 1 0.000049 -3223> 2014-11-08 18:15:34.167119 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/46 crt=5734'8252 inactive NOTIFY] enter Started -3222> 2014-11-08 18:15:34.167126 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/46 crt=5734'8252 inactive NOTIFY] enter Start -3221> 2014-11-08 18:15:34.167131 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/46 crt=5734'8252 inactive NOTIFY] state<Start>: transitioning to Stray -3220> 2014-11-08 18:15:34.167138 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/46 crt=5734'8252 inactive NOTIFY] exit Start 0.000012 0 0.000000 -3219> 2014-11-08 18:15:34.167145 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.2ds2( v 5734'8272 (3458'5271,5734'8272] local-les=5752 n=7696 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/46 crt=5734'8252 inactive NOTIFY] enter Started/Stray -3218> 2014-11-08 18:15:34.167223 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=-1 lpr=5759 pi=4872-5758/49 luod=0'0 crt=5734'8320 active+remapped] exit Started/ReplicaActive/RepNotRecovering 965.445718 27 0.000278 -3217> 2014-11-08 18:15:34.167235 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=-1 lpr=5759 pi=4872-5758/49 luod=0'0 crt=5734'8320 active+remapped] exit Started/ReplicaActive 965.445742 0 0.000000 -3216> 2014-11-08 18:15:34.167244 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=-1 lpr=5759 pi=4872-5758/49 luod=0'0 crt=5734'8320 active+remapped] exit Started 1015.780233 0 0.000000 -3215> 2014-11-08 18:15:34.167252 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=-1 lpr=5759 pi=4872-5758/49 luod=0'0 crt=5734'8320 active+remapped] enter Reset -3214> 2014-11-08 18:15:34.167327 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=4872-5773/50 crt=5734'8320 remapped NOTIFY] exit Reset 0.000075 1 0.000052 -3213> 2014-11-08 18:15:34.167336 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=4872-5773/50 crt=5734'8320 remapped NOTIFY] enter Started -3212> 2014-11-08 18:15:34.167344 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=4872-5773/50 crt=5734'8320 remapped NOTIFY] enter Start -3211> 2014-11-08 18:15:34.167350 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=4872-5773/50 crt=5734'8320 remapped NOTIFY] state<Start>: transitioning to Stray -3210> 2014-11-08 18:15:34.167358 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=4872-5773/50 crt=5734'8320 remapped NOTIFY] exit Start 0.000014 0 0.000000 -3209> 2014-11-08 18:15:34.167366 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s2( v 5734'8320 (3458'5290,5734'8320] lb 57ef0021/1000002aa1a.0000002d/head//4 local-les=0 n=29 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=4872-5773/50 crt=5734'8320 remapped NOTIFY] enter Started/Stray -3208> 2014-11-08 18:15:34.167458 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=1 lpr=5759 pi=4860-5758/52 bft=2(1),5(0),9(2) crt=5700'8298 lcod 5734'8319 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 961.069151 25 0.000105 -3207> 2014-11-08 18:15:34.167471 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=1 lpr=5759 pi=4860-5758/52 bft=2(1),5(0),9(2) crt=5700'8298 lcod 5734'8319 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 965.447489 0 0.000000 -3206> 2014-11-08 18:15:34.167483 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=1 lpr=5759 pi=4860-5758/52 bft=2(1),5(0),9(2) crt=5700'8298 lcod 5734'8319 mlcod 0'0 active+remapped] exit Started/Primary 1015.779961 0 0.000000 -3205> 2014-11-08 18:15:34.167492 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=1 lpr=5759 pi=4860-5758/52 bft=2(1),5(0),9(2) crt=5700'8298 lcod 5734'8319 mlcod 0'0 active+remapped] exit Started 1015.780016 0 0.000000 -3204> 2014-11-08 18:15:34.167500 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5759/5728) [5,2,9]/[2147483647,9,4] r=1 lpr=5759 pi=4860-5758/52 bft=2(1),5(0),9(2) crt=5700'8298 lcod 5734'8319 mlcod 0'0 active+remapped] enter Reset -3203> 2014-11-08 18:15:34.167534 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] exit Reset 0.000034 1 0.000075 -3202> 2014-11-08 18:15:34.167543 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] enter Started -3201> 2014-11-08 18:15:34.167550 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] enter Start -3200> 2014-11-08 18:15:34.167557 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] state<Start>: transitioning to Primary -3199> 2014-11-08 18:15:34.167564 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -3198> 2014-11-08 18:15:34.167572 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] enter Started/Primary -3197> 2014-11-08 18:15:34.167579 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped] enter Started/Primary/Peering -3196> 2014-11-08 18:15:34.167586 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.21s1( v 5734'8320 (3458'5289,5734'8320] local-les=5765 n=7747 ec=195 les/c 5765/4877 5759/5774/5728) [5,2,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4860-5773/53 crt=5700'8298 lcod 5734'8319 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -3195> 2014-11-08 18:15:34.167726 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5721'8088 active] exit Started/ReplicaActive/RepNotRecovering 1661.158186 58 0.000309 -3194> 2014-11-08 18:15:34.167739 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5721'8088 active] exit Started/ReplicaActive 1661.158207 0 0.000000 -3193> 2014-11-08 18:15:34.167747 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5721'8088 active] exit Started 1666.285637 0 0.000000 -3192> 2014-11-08 18:15:34.167754 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5721'8088 active] enter Reset -3191> 2014-11-08 18:15:34.167783 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] exit Reset 0.000029 1 0.000050 -3190> 2014-11-08 18:15:34.167792 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] enter Started -3189> 2014-11-08 18:15:34.167798 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] enter Start -3188> 2014-11-08 18:15:34.167803 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] state<Start>: transitioning to Stray -3187> 2014-11-08 18:15:34.167810 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] exit Start 0.000013 0 0.000000 -3186> 2014-11-08 18:15:34.167817 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] enter Started/Stray -3185> 2014-11-08 18:15:34.167908 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5759/5641) [1,9,5]/[1,2147483647,4] r=-1 lpr=5759 pi=4863-5758/52 luod=0'0 crt=5734'8079 active+remapped] exit Started/ReplicaActive/RepNotRecovering 965.507937 27 0.000274 -3184> 2014-11-08 18:15:34.167919 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5759/5641) [1,9,5]/[1,2147483647,4] r=-1 lpr=5759 pi=4863-5758/52 luod=0'0 crt=5734'8079 active+remapped] exit Started/ReplicaActive 965.507963 0 0.000000 -3183> 2014-11-08 18:15:34.167928 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5759/5641) [1,9,5]/[1,2147483647,4] r=-1 lpr=5759 pi=4863-5758/52 luod=0'0 crt=5734'8079 active+remapped] exit Started 1015.779053 0 0.000000 -3182> 2014-11-08 18:15:34.167949 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5759/5641) [1,9,5]/[1,2147483647,4] r=-1 lpr=5759 pi=4863-5758/52 luod=0'0 crt=5734'8079 active+remapped] enter Reset -3181> 2014-11-08 18:15:34.168018 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5774/5641) [1,9,5]/[1,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/53 crt=5734'8079 remapped NOTIFY] exit Reset 0.000068 1 0.000065 -3180> 2014-11-08 18:15:34.168027 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5774/5641) [1,9,5]/[1,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/53 crt=5734'8079 remapped NOTIFY] enter Started -3179> 2014-11-08 18:15:34.168034 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5774/5641) [1,9,5]/[1,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/53 crt=5734'8079 remapped NOTIFY] enter Start -3178> 2014-11-08 18:15:34.168040 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5774/5641) [1,9,5]/[1,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/53 crt=5734'8079 remapped NOTIFY] state<Start>: transitioning to Stray -3177> 2014-11-08 18:15:34.168048 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5774/5641) [1,9,5]/[1,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/53 crt=5734'8079 remapped NOTIFY] exit Start 0.000014 0 0.000000 -3176> 2014-11-08 18:15:34.168056 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.12s1( v 5734'8079 (3458'5043,5734'8079] lb d06b0012/10000029dfe.00000005/head//4 local-les=0 n=27 ec=195 les/c 5765/4864 5759/5774/5641) [1,9,5]/[1,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/53 crt=5734'8079 remapped NOTIFY] enter Started/Stray -3175> 2014-11-08 18:15:34.168157 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=2936-5750/43 luod=0'0 crt=5734'8200 active] exit Started/ReplicaActive/RepNotRecovering 1665.863548 65 0.001191 -3174> 2014-11-08 18:15:34.168171 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=2936-5750/43 luod=0'0 crt=5734'8200 active] exit Started/ReplicaActive 1665.863571 0 0.000000 -3173> 2014-11-08 18:15:34.168179 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=2936-5750/43 luod=0'0 crt=5734'8200 active] exit Started 1666.285563 0 0.000000 -3172> 2014-11-08 18:15:34.168186 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5751/5751/5751) [4,2,9] r=2 lpr=5751 pi=2936-5750/43 luod=0'0 crt=5734'8200 active] enter Reset -3171> 2014-11-08 18:15:34.168248 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/44 crt=5734'8200 inactive NOTIFY] exit Reset 0.000062 1 0.000050 -3170> 2014-11-08 18:15:34.168257 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/44 crt=5734'8200 inactive NOTIFY] enter Started -3169> 2014-11-08 18:15:34.168262 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/44 crt=5734'8200 inactive NOTIFY] enter Start -3168> 2014-11-08 18:15:34.168268 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/44 crt=5734'8200 inactive NOTIFY] state<Start>: transitioning to Stray -3167> 2014-11-08 18:15:34.168274 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/44 crt=5734'8200 inactive NOTIFY] exit Start 0.000012 0 0.000000 -3166> 2014-11-08 18:15:34.168281 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.14s2( v 5734'8206 (3458'5205,5734'8206] local-les=5752 n=7676 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/44 crt=5734'8200 inactive NOTIFY] enter Started/Stray -3165> 2014-11-08 18:15:34.168382 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=2 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+undersized+degraded+remapped+backfilling] exit Started/Primary/Active/Backfilling 1021.181709 46 0.000229 -3164> 2014-11-08 18:15:34.168395 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=2 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+undersized+degraded+remapped] exit Started/Primary/Active 1664.852992 0 0.000000 -3163> 2014-11-08 18:15:34.168409 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=2 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+remapped] exit Started/Primary 1666.278319 0 0.000000 -3162> 2014-11-08 18:15:34.168419 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=2 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+remapped] exit Started 1666.278755 0 0.000000 -3161> 2014-11-08 18:15:34.168427 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5751/5751/5728) [1,10,4]/[2147483647,9,1] r=1 lpr=5751 pi=4860-5750/30 rops=2 bft=1(0),4(2),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 active+remapped] enter Reset -3160> 2014-11-08 18:15:34.168564 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] exit Reset 0.000137 1 0.000182 -3159> 2014-11-08 18:15:34.168574 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] enter Started -3158> 2014-11-08 18:15:34.168581 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] enter Start -3157> 2014-11-08 18:15:34.168587 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] state<Start>: transitioning to Primary -3156> 2014-11-08 18:15:34.168595 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -3155> 2014-11-08 18:15:34.168603 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] enter Started/Primary -3154> 2014-11-08 18:15:34.168610 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] enter Started/Primary/Peering -3153> 2014-11-08 18:15:34.168617 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -3152> 2014-11-08 18:15:34.168706 7fc5b1b19700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -3151> 2014-11-08 18:15:34.168710 7fc5b1b19700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- osd_alive(want up_thru 5774 have 5774) v1 -- ?+0 0x22585460 con 0x44ed6a0 -3150> 2014-11-08 18:15:34.168856 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5751/5751/5639) [2,9,4]/[2,9,2147483647] r=1 lpr=5751 pi=2951-5750/75 luod=0'0 crt=5734'8349 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1665.846665 64 0.000310 -3149> 2014-11-08 18:15:34.168870 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5751/5751/5639) [2,9,4]/[2,9,2147483647] r=1 lpr=5751 pi=2951-5750/75 luod=0'0 crt=5734'8349 active+remapped] exit Started/ReplicaActive 1665.846687 0 0.000000 -3148> 2014-11-08 18:15:34.168879 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5751/5751/5639) [2,9,4]/[2,9,2147483647] r=1 lpr=5751 pi=2951-5750/75 luod=0'0 crt=5734'8349 active+remapped] exit Started 1666.280309 0 0.000000 -3147> 2014-11-08 18:15:34.168886 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5751/5751/5639) [2,9,4]/[2,9,2147483647] r=1 lpr=5751 pi=2951-5750/75 luod=0'0 crt=5734'8349 active+remapped] enter Reset -3146> 2014-11-08 18:15:34.169016 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=2951-5773/76 crt=5734'8349 inactive NOTIFY] exit Reset 0.000130 1 0.000054 -3145> 2014-11-08 18:15:34.169025 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=2951-5773/76 crt=5734'8349 inactive NOTIFY] enter Started -3144> 2014-11-08 18:15:34.169031 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=2951-5773/76 crt=5734'8349 inactive NOTIFY] enter Start -3143> 2014-11-08 18:15:34.169037 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=2951-5773/76 crt=5734'8349 inactive NOTIFY] state<Start>: transitioning to Stray -3142> 2014-11-08 18:15:34.169043 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=2951-5773/76 crt=5734'8349 inactive NOTIFY] exit Start 0.000012 0 0.000000 -3141> 2014-11-08 18:15:34.169050 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bcs1( v 5734'8351 (3458'5318,5734'8351] local-les=5752 n=7742 ec=195 les/c 5752/5750 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=2951-5773/76 crt=5734'8349 inactive NOTIFY] enter Started/Stray -3140> 2014-11-08 18:15:34.169133 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5751/5751/5639) [2,9,4]/[2,2147483647,10] r=-1 lpr=5751 pi=5453-5750/18 luod=0'0 crt=5734'8156 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1661.159326 57 0.000295 -3139> 2014-11-08 18:15:34.169144 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5751/5751/5639) [2,9,4]/[2,2147483647,10] r=-1 lpr=5751 pi=5453-5750/18 luod=0'0 crt=5734'8156 active+remapped] exit Started/ReplicaActive 1661.159345 0 0.000000 -3138> 2014-11-08 18:15:34.169152 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5751/5751/5639) [2,9,4]/[2,2147483647,10] r=-1 lpr=5751 pi=5453-5750/18 luod=0'0 crt=5734'8156 active+remapped] exit Started 1666.280403 0 0.000000 -3137> 2014-11-08 18:15:34.169160 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5751/5751/5639) [2,9,4]/[2,2147483647,10] r=-1 lpr=5751 pi=5453-5750/18 luod=0'0 crt=5734'8156 active+remapped] enter Reset -3136> 2014-11-08 18:15:34.169202 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5774/5774/5639) [2,9,2147483647]/[2,2147483647,10] r=-1 lpr=5774 pi=5453-5773/19 crt=5734'8156 remapped NOTIFY] exit Reset 0.000042 1 0.000049 -3135> 2014-11-08 18:15:34.169212 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5774/5774/5639) [2,9,2147483647]/[2,2147483647,10] r=-1 lpr=5774 pi=5453-5773/19 crt=5734'8156 remapped NOTIFY] enter Started -3134> 2014-11-08 18:15:34.169219 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5774/5774/5639) [2,9,2147483647]/[2,2147483647,10] r=-1 lpr=5774 pi=5453-5773/19 crt=5734'8156 remapped NOTIFY] enter Start -3133> 2014-11-08 18:15:34.169226 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5774/5774/5639) [2,9,2147483647]/[2,2147483647,10] r=-1 lpr=5774 pi=5453-5773/19 crt=5734'8156 remapped NOTIFY] state<Start>: transitioning to Stray -3132> 2014-11-08 18:15:34.169233 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5774/5774/5639) [2,9,2147483647]/[2,2147483647,10] r=-1 lpr=5774 pi=5453-5773/19 crt=5734'8156 remapped NOTIFY] exit Start 0.000014 0 0.000000 -3131> 2014-11-08 18:15:34.169241 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.bds1( v 5734'8156 (3458'5120,5734'8156] lb af68e1bd/1000002a3d1.0000004a/head//4 local-les=0 n=902 ec=195 les/c 5753/5454 5774/5774/5639) [2,9,2147483647]/[2,2147483647,10] r=-1 lpr=5774 pi=5453-5773/19 crt=5734'8156 remapped NOTIFY] enter Started/Stray -3130> 2014-11-08 18:15:34.169293 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5751/5751/5729) [9,4,2] r=0 lpr=5751 crt=5734'8162 lcod 5734'8163 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1661.582082 62 0.000184 -3129> 2014-11-08 18:15:34.169303 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5751/5751/5729) [9,4,2] r=0 lpr=5751 crt=5734'8162 lcod 5734'8163 mlcod 0'0 active] exit Started/Primary/Active 1665.869493 0 0.000000 -3128> 2014-11-08 18:15:34.169321 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5751/5751/5729) [9,4,2] r=0 lpr=5751 crt=5734'8162 lcod 5734'8163 mlcod 0'0 active] exit Started/Primary 1666.280401 0 0.000000 -3127> 2014-11-08 18:15:34.169328 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5751/5751/5729) [9,4,2] r=0 lpr=5751 crt=5734'8162 lcod 5734'8163 mlcod 0'0 active] exit Started 1666.280433 0 0.000000 -3126> 2014-11-08 18:15:34.169335 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5751/5751/5729) [9,4,2] r=0 lpr=5751 crt=5734'8162 lcod 5734'8163 mlcod 0'0 active] enter Reset -3125> 2014-11-08 18:15:34.169385 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] exit Reset 0.000050 1 0.000093 -3124> 2014-11-08 18:15:34.169394 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] enter Started -3123> 2014-11-08 18:15:34.169410 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] enter Start -3122> 2014-11-08 18:15:34.169416 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3121> 2014-11-08 18:15:34.169423 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] exit Start 0.000013 0 0.000000 -3120> 2014-11-08 18:15:34.169430 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] enter Started/Primary -3119> 2014-11-08 18:15:34.169437 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] enter Started/Primary/Peering -3118> 2014-11-08 18:15:34.169443 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3117> 2014-11-08 18:15:34.169485 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5751/5751/5639) [4,2,9]/[2147483647,2,9] r=2 lpr=5751 pi=2951-5750/70 luod=0'0 crt=5734'8255 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1665.846980 64 0.000334 -3116> 2014-11-08 18:15:34.169495 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5751/5751/5639) [4,2,9]/[2147483647,2,9] r=2 lpr=5751 pi=2951-5750/70 luod=0'0 crt=5734'8255 active+remapped] exit Started/ReplicaActive 1665.846999 0 0.000000 -3115> 2014-11-08 18:15:34.169504 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5751/5751/5639) [4,2,9]/[2147483647,2,9] r=2 lpr=5751 pi=2951-5750/70 luod=0'0 crt=5734'8255 active+remapped] exit Started 1666.279800 0 0.000000 -3114> 2014-11-08 18:15:34.169511 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5751/5751/5639) [4,2,9]/[2147483647,2,9] r=2 lpr=5751 pi=2951-5750/70 luod=0'0 crt=5734'8255 active+remapped] enter Reset -3113> 2014-11-08 18:15:34.169562 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5774/5774/5639) [2147483647,2,9] r=2 lpr=5774 pi=2951-5773/71 crt=5734'8255 inactive NOTIFY] exit Reset 0.000051 1 0.000048 -3112> 2014-11-08 18:15:34.169571 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5774/5774/5639) [2147483647,2,9] r=2 lpr=5774 pi=2951-5773/71 crt=5734'8255 inactive NOTIFY] enter Started -3111> 2014-11-08 18:15:34.169577 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5774/5774/5639) [2147483647,2,9] r=2 lpr=5774 pi=2951-5773/71 crt=5734'8255 inactive NOTIFY] enter Start -3110> 2014-11-08 18:15:34.169583 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5774/5774/5639) [2147483647,2,9] r=2 lpr=5774 pi=2951-5773/71 crt=5734'8255 inactive NOTIFY] state<Start>: transitioning to Stray -3109> 2014-11-08 18:15:34.169589 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5774/5774/5639) [2147483647,2,9] r=2 lpr=5774 pi=2951-5773/71 crt=5734'8255 inactive NOTIFY] exit Start 0.000012 0 0.000000 -3108> 2014-11-08 18:15:34.169596 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b2s2( v 5734'8260 (3458'5229,5734'8260] local-les=5752 n=7674 ec=195 les/c 5752/5750 5774/5774/5639) [2147483647,2,9] r=2 lpr=5774 pi=2951-5773/71 crt=5734'8255 inactive NOTIFY] enter Started/Stray -3107> 2014-11-08 18:15:34.169823 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 331 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x22acad00 con 0x1c7911e0 -3106> 2014-11-08 18:15:34.169826 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- osd_map(5774..5774 src has 4761..5774) v3 -- ?+0 0x22acc800 con 0x1c5c2680 -3105> 2014-11-08 18:15:34.169845 7fc5b9b29700 3 osd.9 5774 handle_osd_map epochs [5774,5774], i have 5774, src has [4761,5774] -3104> 2014-11-08 18:15:34.169894 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.24s1(11),4.12s1(53) epoch 5774) v5 -- ?+0 0x2515e180 con 0x1c5c2680 -3103> 2014-11-08 18:15:34.170027 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- osd_map(5774..5774 src has 4761..5774) v3 -- ?+0 0x22acf980 con 0x1c5c0580 -3102> 2014-11-08 18:15:34.170032 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 332 ==== pg_notify(4.15s1(28) epoch 5774) v5 ==== 2427+0+0 (3285636965 0 0) 0x22451c20 con 0x1c7911e0 -3101> 2014-11-08 18:15:34.170042 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.169963, event: header_read, op: pg_notify(4.15s1(28) epoch 5774) -3100> 2014-11-08 18:15:34.170050 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.169964, event: throttled, op: pg_notify(4.15s1(28) epoch 5774) -3099> 2014-11-08 18:15:34.170052 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.170019, event: all_read, op: pg_notify(4.15s1(28) epoch 5774) -3098> 2014-11-08 18:15:34.170055 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.170040, event: dispatched, op: pg_notify(4.15s1(28) epoch 5774) -3097> 2014-11-08 18:15:34.170057 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.170057, event: waiting_for_osdmap, op: pg_notify(4.15s1(28) epoch 5774) -3096> 2014-11-08 18:15:34.170062 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.170062, event: started, op: pg_notify(4.15s1(28) epoch 5774) -3095> 2014-11-08 18:15:34.170074 7fc5b9b29700 5 -- op tracker -- seq: 47228, time: 2014-11-08 18:15:34.170074, event: done, op: pg_notify(4.15s1(28) epoch 5774) -3094> 2014-11-08 18:15:34.170087 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.2as2(35),4.2ds2(46),4.14s2(44) epoch 5774) v5 -- ?+0 0x2515cb00 con 0x1c5c0580 -3093> 2014-11-08 18:15:34.170292 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.21s2(50) epoch 5774) v5 -- ?+0 0x251581e0 con 0x42c86e0 -3092> 2014-11-08 18:15:34.170330 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_query(4.15s0,4.15s2 epoch 5774) v3 -- ?+0 0x2515fbc0 con 0x1c5c2680 -3091> 2014-11-08 18:15:34.170331 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.21s2(50) epoch 5774) v5 ==== 0+0+0 (0 0 0) 0x251581e0 con 0x42c86e0 -3090> 2014-11-08 18:15:34.170346 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 2014-11-08 18:15:34.170316, event: header_read, op: pg_notify(4.21s2(50) epoch 5774) -3089> 2014-11-08 18:15:34.170352 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 0.000000, event: throttled, op: pg_notify(4.21s2(50) epoch 5774) -3088> 2014-11-08 18:15:34.170354 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 0.000000, event: all_read, op: pg_notify(4.21s2(50) epoch 5774) -3087> 2014-11-08 18:15:34.170356 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 2014-11-08 18:15:34.170345, event: dispatched, op: pg_notify(4.21s2(50) epoch 5774) -3086> 2014-11-08 18:15:34.170359 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 2014-11-08 18:15:34.170359, event: waiting_for_osdmap, op: pg_notify(4.21s2(50) epoch 5774) -3085> 2014-11-08 18:15:34.170363 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 2014-11-08 18:15:34.170362, event: started, op: pg_notify(4.21s2(50) epoch 5774) -3084> 2014-11-08 18:15:34.170373 7fc5b9b29700 5 -- op tracker -- seq: 47229, time: 2014-11-08 18:15:34.170373, event: done, op: pg_notify(4.21s2(50) epoch 5774) -3083> 2014-11-08 18:15:34.170384 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_query(4.21s1,4.28s1 epoch 5774) v3 -- ?+0 0x23c35820 con 0x1c5c0580 -3082> 2014-11-08 18:15:34.170446 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- osd_map(5774..5774 src has 4761..5774) v3 -- ?+0 0x22acf080 con 0x1e50aec0 -3081> 2014-11-08 18:15:34.170502 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_query(4.21s0 epoch 5774) v3 -- ?+0 0x22e592c0 con 0x1e50aec0 -3080> 2014-11-08 18:15:34.170544 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_query(4.21s2 epoch 5774) v3 -- ?+0 0x234e7da0 con 0x42c86e0 -3079> 2014-11-08 18:15:34.170567 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_query(4.21s2 epoch 5774) v3 ==== 0+0+0 (0 0 0) 0x234e7da0 con 0x42c86e0 -3078> 2014-11-08 18:15:34.170577 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 2014-11-08 18:15:34.170556, event: header_read, op: pg_query(4.21s2 epoch 5774) -3077> 2014-11-08 18:15:34.170583 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 0.000000, event: throttled, op: pg_query(4.21s2 epoch 5774) -3076> 2014-11-08 18:15:34.170586 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 0.000000, event: all_read, op: pg_query(4.21s2 epoch 5774) -3075> 2014-11-08 18:15:34.170590 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 2014-11-08 18:15:34.170576, event: dispatched, op: pg_query(4.21s2 epoch 5774) -3074> 2014-11-08 18:15:34.170594 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 2014-11-08 18:15:34.170594, event: waiting_for_osdmap, op: pg_query(4.21s2 epoch 5774) -3073> 2014-11-08 18:15:34.170581 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- osd_map(5774..5774 src has 4761..5774) v3 -- ?+0 0x22acc5c0 con 0x1c7911e0 -3072> 2014-11-08 18:15:34.170601 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 2014-11-08 18:15:34.170601, event: started, op: pg_query(4.21s2 epoch 5774) -3071> 2014-11-08 18:15:34.170609 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_query(4.15s1 epoch 5774) v3 -- ?+0 0x22d52b20 con 0x1c7911e0 -3070> 2014-11-08 18:15:34.170654 7fc5b9b29700 5 -- op tracker -- seq: 47230, time: 2014-11-08 18:15:34.170654, event: done, op: pg_query(4.21s2 epoch 5774) -3069> 2014-11-08 18:15:34.170791 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8319 lcod 5734'8325 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1661.164817 59 0.000186 -3068> 2014-11-08 18:15:34.170813 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8319 lcod 5734'8325 mlcod 0'0 active] exit Started/Primary/Active 1665.872524 0 0.000000 -3067> 2014-11-08 18:15:34.170830 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8319 lcod 5734'8325 mlcod 0'0 active] exit Started/Primary 1666.278642 0 0.000000 -3066> 2014-11-08 18:15:34.170844 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8319 lcod 5734'8325 mlcod 0'0 active] exit Started 1666.278699 0 0.000000 -3065> 2014-11-08 18:15:34.170858 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8319 lcod 5734'8325 mlcod 0'0 active] enter Reset -3064> 2014-11-08 18:15:34.170912 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] exit Reset 0.000054 1 0.000120 -3063> 2014-11-08 18:15:34.170967 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] enter Started -3062> 2014-11-08 18:15:34.170982 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] enter Start -3061> 2014-11-08 18:15:34.170995 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3060> 2014-11-08 18:15:34.171010 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] exit Start 0.000027 0 0.000000 -3059> 2014-11-08 18:15:34.171024 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] enter Started/Primary -3058> 2014-11-08 18:15:34.171037 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] enter Started/Primary/Peering -3057> 2014-11-08 18:15:34.171051 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3056> 2014-11-08 18:15:34.171169 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8285 lcod 5734'8294 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1661.584160 62 0.000162 -3055> 2014-11-08 18:15:34.171188 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8285 lcod 5734'8294 mlcod 0'0 active] exit Started/Primary/Active 1665.872541 0 0.000000 -3054> 2014-11-08 18:15:34.171203 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8285 lcod 5734'8294 mlcod 0'0 active] exit Started/Primary 1666.278684 0 0.000000 -3053> 2014-11-08 18:15:34.171217 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8285 lcod 5734'8294 mlcod 0'0 active] exit Started 1666.278737 0 0.000000 -3052> 2014-11-08 18:15:34.171240 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5751/5751/5731) [9,4,2] r=0 lpr=5751 crt=5734'8285 lcod 5734'8294 mlcod 0'0 active] enter Reset -3051> 2014-11-08 18:15:34.170654 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5751/5751/5639) [1,9,4]/[2147483647,2,4] r=-1 lpr=5751 pi=5432-5750/21 luod=0'0 crt=5734'8276 active+remapped] exit Started/ReplicaActive/RepRecovering 1025.623853 49 0.000109 -3050> 2014-11-08 18:15:34.171261 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5751/5751/5639) [1,9,4]/[2147483647,2,4] r=-1 lpr=5751 pi=5432-5750/21 luod=0'0 crt=5734'8276 active+remapped] exit Started/ReplicaActive 1665.282033 0 0.000000 -3049> 2014-11-08 18:15:34.171331 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5751/5751/5639) [1,9,4]/[2147483647,2,4] r=-1 lpr=5751 pi=5432-5750/21 luod=0'0 crt=5734'8276 active+remapped] exit Started 1666.280573 0 0.000000 -3048> 2014-11-08 18:15:34.171347 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5751/5751/5639) [1,9,4]/[2147483647,2,4] r=-1 lpr=5751 pi=5432-5750/21 luod=0'0 crt=5734'8276 active+remapped] enter Reset -3047> 2014-11-08 18:15:34.171457 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5774/5774/5639) [1,9,2147483647]/[2147483647,2,2147483647] r=-1 lpr=5774 pi=5432-5773/22 crt=5734'8276 remapped NOTIFY] exit Reset 0.000110 1 0.000724 -3046> 2014-11-08 18:15:34.171312 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] exit Reset 0.000071 1 0.000132 -3045> 2014-11-08 18:15:34.171490 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] enter Started -3044> 2014-11-08 18:15:34.171504 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] enter Start -3043> 2014-11-08 18:15:34.171517 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] state<Start>: transitioning to Primary -3042> 2014-11-08 18:15:34.171531 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] exit Start 0.000027 0 0.000000 -3041> 2014-11-08 18:15:34.171545 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] enter Started/Primary -3040> 2014-11-08 18:15:34.171558 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] enter Started/Primary/Peering -3039> 2014-11-08 18:15:34.171571 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -3038> 2014-11-08 18:15:34.171626 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8075 active] exit Started/ReplicaActive/RepNotRecovering 1661.157463 58 0.000370 -3037> 2014-11-08 18:15:34.171641 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5774/5774/5639) [1,9,2147483647]/[2147483647,2,2147483647] r=-1 lpr=5774 pi=5432-5773/22 crt=5734'8276 remapped NOTIFY] enter Started -3036> 2014-11-08 18:15:34.171660 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5774/5774/5639) [1,9,2147483647]/[2147483647,2,2147483647] r=-1 lpr=5774 pi=5432-5773/22 crt=5734'8276 remapped NOTIFY] enter Start -3035> 2014-11-08 18:15:34.171674 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5774/5774/5639) [1,9,2147483647]/[2147483647,2,2147483647] r=-1 lpr=5774 pi=5432-5773/22 crt=5734'8276 remapped NOTIFY] state<Start>: transitioning to Stray -3034> 2014-11-08 18:15:34.171755 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5774/5774/5639) [1,9,2147483647]/[2147483647,2,2147483647] r=-1 lpr=5774 pi=5432-5773/22 crt=5734'8276 remapped NOTIFY] exit Start 0.000094 0 0.000000 -3033> 2014-11-08 18:15:34.171782 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.b5s1( v 5734'8276 (3458'5233,5734'8276] lb 70f90db5/1000002b0b9.00000062/head//4 local-les=0 n=6248 ec=195 les/c 5752/5433 5774/5774/5639) [1,9,2147483647]/[2147483647,2,2147483647] r=-1 lpr=5774 pi=5432-5773/22 crt=5734'8276 remapped NOTIFY] enter Started/Stray -3032> 2014-11-08 18:15:34.171768 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8075 active] exit Started/ReplicaActive 1661.157616 0 0.000000 -3031> 2014-11-08 18:15:34.171804 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8075 active] exit Started 1666.278323 0 0.000000 -3030> 2014-11-08 18:15:34.171907 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8075 active] enter Reset -3029> 2014-11-08 18:15:34.171945 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] exit Reset 0.000038 1 0.000311 -3028> 2014-11-08 18:15:34.171960 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] enter Started -3027> 2014-11-08 18:15:34.171973 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] enter Start -3026> 2014-11-08 18:15:34.171985 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] state<Start>: transitioning to Stray -3025> 2014-11-08 18:15:34.171998 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] exit Start 0.000025 0 0.000000 -3024> 2014-11-08 18:15:34.171894 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.ds1(11),4.7s2(24) epoch 5774) v5 -- ?+0 0x2127aee0 con 0x1c5c2680 -3023> 2014-11-08 18:15:34.172022 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] enter Started/Stray -3022> 2014-11-08 18:15:34.172174 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5453-5750/21 luod=0'0 crt=0'0 active] exit Started/ReplicaActive/RepNotRecovering 1665.856711 65 0.000380 -3021> 2014-11-08 18:15:34.172111 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.6s2(24),4.bcs1(76),4.bds1(19),4.b2s2(71),4.b5s1(22) epoch 5774) v5 -- ?+0 0x2127ad00 con 0x1c5c0580 -3020> 2014-11-08 18:15:34.172402 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_query(4.9s1,4.cs1,4.es2 epoch 5774) v3 -- ?+0 0x2127ab20 con 0x1c5c2680 -3019> 2014-11-08 18:15:34.172454 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_query(4.5s1,4.as1,4.b3s2,4.bfs2 epoch 5774) v3 -- ?+0 0x2127a940 con 0x1c5c0580 -3018> 2014-11-08 18:15:34.172517 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_query(4.bfs2 epoch 5774) v3 -- ?+0 0x2127a760 con 0x1e50aec0 -3017> 2014-11-08 18:15:34.172591 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8105 active] exit Started/ReplicaActive/RepNotRecovering 1661.155784 58 0.000286 -3016> 2014-11-08 18:15:34.172606 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5453-5750/21 luod=0'0 crt=0'0 active] exit Started/ReplicaActive 1665.857163 0 0.000000 -3015> 2014-11-08 18:15:34.172626 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5453-5750/21 luod=0'0 crt=0'0 active] exit Started 1666.278957 0 0.000000 -3014> 2014-11-08 18:15:34.172640 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=5453-5750/21 luod=0'0 crt=0'0 active] enter Reset -3013> 2014-11-08 18:15:34.172692 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5453-5773/22 crt=0'0 inactive NOTIFY] exit Reset 0.000051 1 0.000501 -3012> 2014-11-08 18:15:34.172704 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8105 active] exit Started/ReplicaActive 1661.155911 0 0.000000 -3011> 2014-11-08 18:15:34.172724 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8105 active] exit Started 1666.274700 0 0.000000 -3010> 2014-11-08 18:15:34.172738 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5751/5751/5641) [1,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8105 active] enter Reset -3009> 2014-11-08 18:15:34.172774 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] exit Reset 0.000036 1 0.000177 -3008> 2014-11-08 18:15:34.172786 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5453-5773/22 crt=0'0 inactive NOTIFY] enter Started -3007> 2014-11-08 18:15:34.172804 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5453-5773/22 crt=0'0 inactive NOTIFY] enter Start -3006> 2014-11-08 18:15:34.172816 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5453-5773/22 crt=0'0 inactive NOTIFY] state<Start>: transitioning to Stray -3005> 2014-11-08 18:15:34.172839 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5453-5773/22 crt=0'0 inactive NOTIFY] exit Start 0.000035 0 0.000000 -3004> 2014-11-08 18:15:34.172826 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] enter Started -3003> 2014-11-08 18:15:34.172870 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] enter Start -3002> 2014-11-08 18:15:34.172882 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9cs2( v 5734'8174 (3458'5154,5734'8174] local-les=5752 n=7719 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=5453-5773/22 crt=0'0 inactive NOTIFY] enter Started/Stray -3001> 2014-11-08 18:15:34.172898 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] state<Start>: transitioning to Stray -3000> 2014-11-08 18:15:34.173482 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] exit Start 0.000612 0 0.000000 -2999> 2014-11-08 18:15:34.173500 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] enter Started/Stray -2998> 2014-11-08 18:15:34.172956 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=2951-5750/41 luod=0'0 crt=5734'8359 active] exit Started/ReplicaActive/RepNotRecovering 1661.156417 58 0.000326 -2997> 2014-11-08 18:15:34.173565 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=2951-5750/41 luod=0'0 crt=5734'8359 active] exit Started/ReplicaActive 1661.157040 0 0.000000 -2996> 2014-11-08 18:15:34.173581 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=2951-5750/41 luod=0'0 crt=5734'8359 active] exit Started 1666.279137 0 0.000000 -2995> 2014-11-08 18:15:34.173595 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=2951-5750/41 luod=0'0 crt=5734'8359 active] enter Reset -2994> 2014-11-08 18:15:34.174031 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12086 ==== pg_notify(4.15s0(28),4.15s2(83) epoch 5774) v5 ==== 8067+0+0 (2882881186 0 0) 0x2127ab20 con 0x1c5c2680 -2993> 2014-11-08 18:15:34.174050 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.173724, event: header_read, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2992> 2014-11-08 18:15:34.174060 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.173724, event: throttled, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2991> 2014-11-08 18:15:34.174070 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.173968, event: all_read, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2990> 2014-11-08 18:15:34.174079 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.174047, event: dispatched, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2989> 2014-11-08 18:15:34.174089 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.174089, event: waiting_for_osdmap, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2988> 2014-11-08 18:15:34.174100 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.174100, event: started, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2987> 2014-11-08 18:15:34.174125 7fc5b9b29700 5 -- op tracker -- seq: 47231, time: 2014-11-08 18:15:34.174125, event: done, op: pg_notify(4.15s0(28),4.15s2(83) epoch 5774) -2986> 2014-11-08 18:15:34.178733 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8644 ==== pg_notify(4.21s1(50),4.28s1(7) epoch 5774) v5 ==== 4861+0+0 (3470733096 0 0) 0x2127a940 con 0x1c5c0580 -2985> 2014-11-08 18:15:34.178758 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.174248, event: header_read, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2984> 2014-11-08 18:15:34.178774 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.174248, event: throttled, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2983> 2014-11-08 18:15:34.178783 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.174313, event: all_read, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2982> 2014-11-08 18:15:34.178792 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.178754, event: dispatched, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2981> 2014-11-08 18:15:34.178802 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.178802, event: waiting_for_osdmap, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2980> 2014-11-08 18:15:34.178814 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.178814, event: started, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2979> 2014-11-08 18:15:34.178843 7fc5b9b29700 5 -- op tracker -- seq: 47232, time: 2014-11-08 18:15:34.178843, event: done, op: pg_notify(4.21s1(50),4.28s1(7) epoch 5774) -2978> 2014-11-08 18:15:34.178868 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12087 ==== pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) v5 ==== 6666+0+0 (53834151 0 0) 0x2127aee0 con 0x1c5c2680 -2977> 2014-11-08 18:15:34.178883 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.174987, event: header_read, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2976> 2014-11-08 18:15:34.178893 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.174988, event: throttled, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2975> 2014-11-08 18:15:34.178902 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.175079, event: all_read, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2974> 2014-11-08 18:15:34.178911 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.178882, event: dispatched, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2973> 2014-11-08 18:15:34.178920 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.178920, event: waiting_for_osdmap, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2972> 2014-11-08 18:15:34.179318 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.179318, event: started, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2971> 2014-11-08 18:15:34.179355 7fc5b9b29700 5 -- op tracker -- seq: 47233, time: 2014-11-08 18:15:34.179355, event: done, op: pg_notify(4.9s1(7),4.cs1(34),4.es2(34) epoch 5774) -2970> 2014-11-08 18:15:34.179378 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8645 ==== pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) v5 ==== 8845+0+0 (1419735545 0 0) 0x2127ad00 con 0x1c5c0580 -2969> 2014-11-08 18:15:34.179394 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.176174, event: header_read, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2968> 2014-11-08 18:15:34.179404 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.176175, event: throttled, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2967> 2014-11-08 18:15:34.179413 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.176315, event: all_read, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2966> 2014-11-08 18:15:34.179423 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.179393, event: dispatched, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2965> 2014-11-08 18:15:34.179432 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.179432, event: waiting_for_osdmap, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2964> 2014-11-08 18:15:34.179443 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.179443, event: started, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2963> 2014-11-08 18:15:34.179480 7fc5b9b29700 5 -- op tracker -- seq: 47234, time: 2014-11-08 18:15:34.179480, event: done, op: pg_notify(4.5s1(7),4.as1(7),4.b3s2(44),4.bfs2(41) epoch 5774) -2962> 2014-11-08 18:15:34.179593 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] exit Reset 0.005998 1 0.000671 -2961> 2014-11-08 18:15:34.179614 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] enter Started -2960> 2014-11-08 18:15:34.179628 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] enter Start -2959> 2014-11-08 18:15:34.179641 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] state<Start>: transitioning to Stray -2958> 2014-11-08 18:15:34.179655 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] exit Start 0.000027 0 0.000000 -2957> 2014-11-08 18:15:34.179670 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] enter Started/Stray -2956> 2014-11-08 18:15:34.179786 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'8302 active] exit Started/ReplicaActive/RepNotRecovering 1665.873468 65 0.000276 -2955> 2014-11-08 18:15:34.180349 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'8302 active] exit Started/ReplicaActive 1665.874039 0 0.000000 -2954> 2014-11-08 18:15:34.180370 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'8302 active] exit Started 1666.284660 0 0.000000 -2953> 2014-11-08 18:15:34.180386 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'8302 active] enter Reset -2952> 2014-11-08 18:15:34.180435 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] exit Reset 0.000049 1 0.000645 -2951> 2014-11-08 18:15:34.180451 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] enter Started -2950> 2014-11-08 18:15:34.180464 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] enter Start -2949> 2014-11-08 18:15:34.180477 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2948> 2014-11-08 18:15:34.180492 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] exit Start 0.000028 0 0.000000 -2947> 2014-11-08 18:15:34.180507 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] enter Started/Primary -2946> 2014-11-08 18:15:34.180520 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] enter Started/Primary/Peering -2945> 2014-11-08 18:15:34.180533 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2944> 2014-11-08 18:15:34.180626 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5721'8087 lcod 5734'8110 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1661.174612 59 0.000164 -2943> 2014-11-08 18:15:34.180729 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5721'8087 lcod 5734'8110 mlcod 0'0 active] exit Started/Primary/Active 1665.877428 0 0.000000 -2942> 2014-11-08 18:15:34.180748 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5721'8087 lcod 5734'8110 mlcod 0'0 active] exit Started/Primary 1666.284425 0 0.000000 -2941> 2014-11-08 18:15:34.180762 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5721'8087 lcod 5734'8110 mlcod 0'0 active] exit Started 1666.284486 0 0.000000 -2940> 2014-11-08 18:15:34.180776 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5751/5751/5730) [9,2,4] r=0 lpr=5751 crt=5721'8087 lcod 5734'8110 mlcod 0'0 active] enter Reset -2939> 2014-11-08 18:15:34.180820 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] exit Reset 0.000044 1 0.000194 -2938> 2014-11-08 18:15:34.180851 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] enter Started -2937> 2014-11-08 18:15:34.180865 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] enter Start -2936> 2014-11-08 18:15:34.180877 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2935> 2014-11-08 18:15:34.180897 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] exit Start 0.000033 0 0.000000 -2934> 2014-11-08 18:15:34.181317 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] enter Started/Primary -2933> 2014-11-08 18:15:34.181334 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 333 ==== pg_query(4.91s1 epoch 5774) v3 ==== 144+0+0 (1467794163 0 0) 0x22d52b20 con 0x1c7911e0 -2932> 2014-11-08 18:15:34.181353 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181272, event: header_read, op: pg_query(4.91s1 epoch 5774) -2931> 2014-11-08 18:15:34.181364 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181273, event: throttled, op: pg_query(4.91s1 epoch 5774) -2930> 2014-11-08 18:15:34.181372 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181313, event: all_read, op: pg_query(4.91s1 epoch 5774) -2929> 2014-11-08 18:15:34.181381 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181351, event: dispatched, op: pg_query(4.91s1 epoch 5774) -2928> 2014-11-08 18:15:34.181390 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181390, event: waiting_for_osdmap, op: pg_query(4.91s1 epoch 5774) -2927> 2014-11-08 18:15:34.181401 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181401, event: started, op: pg_query(4.91s1 epoch 5774) -2926> 2014-11-08 18:15:34.181418 7fc5b9b29700 5 -- op tracker -- seq: 47235, time: 2014-11-08 18:15:34.181418, event: done, op: pg_query(4.91s1 epoch 5774) -2925> 2014-11-08 18:15:34.181433 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] enter Started/Primary/Peering -2924> 2014-11-08 18:15:34.181448 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2923> 2014-11-08 18:15:34.181513 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5751/5751/5751) [4,9,1]/[2147483647,4,10] r=-1 lpr=5751 pi=4875-5750/29 luod=0'0 crt=5734'8180 active+remapped] exit Started/ReplicaActive/RepRecovering 1025.976349 49 0.000096 -2922> 2014-11-08 18:15:34.181592 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5751/5751/5751) [4,9,1]/[2147483647,4,10] r=-1 lpr=5751 pi=4875-5750/29 luod=0'0 crt=5734'8180 active+remapped] exit Started/ReplicaActive 1664.846639 0 0.000000 -2921> 2014-11-08 18:15:34.181610 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5751/5751/5751) [4,9,1]/[2147483647,4,10] r=-1 lpr=5751 pi=4875-5750/29 luod=0'0 crt=5734'8180 active+remapped] exit Started 1666.285133 0 0.000000 -2920> 2014-11-08 18:15:34.181625 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5751/5751/5751) [4,9,1]/[2147483647,4,10] r=-1 lpr=5751 pi=4875-5750/29 luod=0'0 crt=5734'8180 active+remapped] enter Reset -2919> 2014-11-08 18:15:34.181732 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5774/5774/5774) [2147483647,9,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=4875-5773/30 crt=5734'8180 remapped NOTIFY] exit Reset 0.000106 1 0.000146 -2918> 2014-11-08 18:15:34.181749 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5774/5774/5774) [2147483647,9,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=4875-5773/30 crt=5734'8180 remapped NOTIFY] enter Started -2917> 2014-11-08 18:15:34.181764 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5774/5774/5774) [2147483647,9,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=4875-5773/30 crt=5734'8180 remapped NOTIFY] enter Start -2916> 2014-11-08 18:15:34.181788 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5774/5774/5774) [2147483647,9,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=4875-5773/30 crt=5734'8180 remapped NOTIFY] state<Start>: transitioning to Stray -2915> 2014-11-08 18:15:34.181803 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5774/5774/5774) [2147483647,9,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=4875-5773/30 crt=5734'8180 remapped NOTIFY] exit Start 0.000040 0 0.000000 -2914> 2014-11-08 18:15:34.181818 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.91s1( v 5734'8180 (3458'5141,5734'8180] lb eaccf291/1000001b4fd.00000128/head//4 local-les=0 n=1436 ec=195 les/c 5752/4877 5774/5774/5774) [2147483647,9,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=4875-5773/30 crt=5734'8180 remapped NOTIFY] enter Started/Stray -2913> 2014-11-08 18:15:34.181891 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8203 active] exit Started/ReplicaActive/RepNotRecovering 1665.866085 65 0.000375 -2912> 2014-11-08 18:15:34.182385 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8203 active] exit Started/ReplicaActive 1665.866587 0 0.000000 -2911> 2014-11-08 18:15:34.182402 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8203 active] exit Started 1666.285440 0 0.000000 -2910> 2014-11-08 18:15:34.182417 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'8203 active] enter Reset -2909> 2014-11-08 18:15:34.182473 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8203 inactive NOTIFY] exit Reset 0.000055 1 0.000559 -2908> 2014-11-08 18:15:34.182489 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8203 inactive NOTIFY] enter Started -2907> 2014-11-08 18:15:34.182502 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8203 inactive NOTIFY] enter Start -2906> 2014-11-08 18:15:34.182514 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8203 inactive NOTIFY] state<Start>: transitioning to Stray -2905> 2014-11-08 18:15:34.182528 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8203 inactive NOTIFY] exit Start 0.000027 0 0.000000 -2904> 2014-11-08 18:15:34.182543 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.94s1( v 5734'8211 (3458'5210,5734'8211] local-les=5752 n=7635 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8203 inactive NOTIFY] enter Started/Stray -2903> 2014-11-08 18:15:34.182666 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'8116 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1665.207900 64 0.000431 -2902> 2014-11-08 18:15:34.182747 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'8116 active+remapped] exit Started/ReplicaActive 1665.207993 0 0.000000 -2901> 2014-11-08 18:15:34.182764 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'8116 active+remapped] exit Started 1666.285358 0 0.000000 -2900> 2014-11-08 18:15:34.182779 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'8116 active+remapped] enter Reset -2899> 2014-11-08 18:15:34.183665 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] exit Reset 0.000885 1 0.000997 -2898> 2014-11-08 18:15:34.183682 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] enter Started -2897> 2014-11-08 18:15:34.183712 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] enter Start -2896> 2014-11-08 18:15:34.183725 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] state<Start>: transitioning to Primary -2895> 2014-11-08 18:15:34.183740 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] exit Start 0.000029 0 0.000000 -2894> 2014-11-08 18:15:34.183756 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] enter Started/Primary -2893> 2014-11-08 18:15:34.183770 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped] enter Started/Primary/Peering -2892> 2014-11-08 18:15:34.183783 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs1( v 5734'8119 (3458'5109,5734'8119] local-les=5752 n=7676 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'8116 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -2891> 2014-11-08 18:15:34.183986 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=-1 lpr=5751 pi=5432-5750/20 luod=0'0 crt=5734'8119 active+remapped] exit Started/ReplicaActive/RepRecovering 1025.981052 49 0.000257 -2890> 2014-11-08 18:15:34.184007 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=-1 lpr=5751 pi=5432-5750/20 luod=0'0 crt=5734'8119 active+remapped] exit Started/ReplicaActive 1665.208848 0 0.000000 -2889> 2014-11-08 18:15:34.184023 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=-1 lpr=5751 pi=5432-5750/20 luod=0'0 crt=5734'8119 active+remapped] exit Started 1666.286271 0 0.000000 -2888> 2014-11-08 18:15:34.184039 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5751/5751/5751) [4,1,9]/[4,9,2147483647] r=-1 lpr=5751 pi=5432-5750/20 luod=0'0 crt=5734'8119 active+remapped] enter Reset -2887> 2014-11-08 18:15:34.184087 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=5432-5773/21 crt=5734'8119 remapped NOTIFY] exit Reset 0.000048 1 0.000085 -2886> 2014-11-08 18:15:34.184104 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=5432-5773/21 crt=5734'8119 remapped NOTIFY] enter Started -2885> 2014-11-08 18:15:34.184119 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=5432-5773/21 crt=5734'8119 remapped NOTIFY] enter Start -2884> 2014-11-08 18:15:34.184132 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=5432-5773/21 crt=5734'8119 remapped NOTIFY] state<Start>: transitioning to Stray -2883> 2014-11-08 18:15:34.184148 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=5432-5773/21 crt=5734'8119 remapped NOTIFY] exit Start 0.000030 0 0.000000 -2882> 2014-11-08 18:15:34.184163 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.8fs2( v 5734'8119 (3458'5110,5734'8119] lb 846d678f/1000002afc4.00000068/head//4 local-les=0 n=3597 ec=195 les/c 5752/5433 5774/5774/5774) [2147483647,1,9]/[2147483647,9,2147483647] r=-1 lpr=5774 pi=5432-5773/21 crt=5734'8119 remapped NOTIFY] enter Started/Stray -2881> 2014-11-08 18:15:34.184693 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.99s1(11),4.9fs2(42) epoch 5774) v5 -- ?+0 0x22d514a0 con 0x1c5c2680 -2880> 2014-11-08 18:15:34.184835 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.9cs2(22),4.94s1(11) epoch 5774) v5 -- ?+0 0x22d51c20 con 0x1c5c0580 -2879> 2014-11-08 18:15:34.184934 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_notify(4.a9s0(45) epoch 5774) v5 -- ?+0 0x22348000 con 0x1e50aec0 -2878> 2014-11-08 18:15:34.185038 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.8fs2(21) epoch 5774) v5 -- ?+0 0x22348780 con 0x42c86e0 -2877> 2014-11-08 18:15:34.185066 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.8fs2(21) epoch 5774) v5 ==== 0+0+0 (0 0 0) 0x22348780 con 0x42c86e0 -2876> 2014-11-08 18:15:34.185081 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 2014-11-08 18:15:34.185051, event: header_read, op: pg_notify(4.8fs2(21) epoch 5774) -2875> 2014-11-08 18:15:34.185092 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 0.000000, event: throttled, op: pg_notify(4.8fs2(21) epoch 5774) -2874> 2014-11-08 18:15:34.185099 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 0.000000, event: all_read, op: pg_notify(4.8fs2(21) epoch 5774) -2873> 2014-11-08 18:15:34.185107 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 2014-11-08 18:15:34.185079, event: dispatched, op: pg_notify(4.8fs2(21) epoch 5774) -2872> 2014-11-08 18:15:34.185116 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 2014-11-08 18:15:34.185116, event: waiting_for_osdmap, op: pg_notify(4.8fs2(21) epoch 5774) -2871> 2014-11-08 18:15:34.185127 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 2014-11-08 18:15:34.185127, event: started, op: pg_notify(4.8fs2(21) epoch 5774) -2870> 2014-11-08 18:15:34.185147 7fc5b9b29700 5 -- op tracker -- seq: 47236, time: 2014-11-08 18:15:34.185147, event: done, op: pg_notify(4.8fs2(21) epoch 5774) -2869> 2014-11-08 18:15:34.185166 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.91s1(30) epoch 5774) v5 -- ?+0 0x22349e00 con 0x1c7911e0 -2868> 2014-11-08 18:15:34.185283 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_query(4.8fs1 epoch 5774) v3 -- ?+0 0x2234a1c0 con 0x1c5c2680 -2867> 2014-11-08 18:15:34.185331 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_query(4.93s1,4.98s2,4.9ds2,4.a0s2 epoch 5774) v3 -- ?+0 0x2234a760 con 0x1c5c0580 -2866> 2014-11-08 18:15:34.185384 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_query(4.8fs2 epoch 5774) v3 -- ?+0 0x2234a940 con 0x42c86e0 -2865> 2014-11-08 18:15:34.185831 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_query(4.8fs2 epoch 5774) v3 ==== 0+0+0 (0 0 0) 0x2234a940 con 0x42c86e0 -2864> 2014-11-08 18:15:34.185847 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 2014-11-08 18:15:34.185396, event: header_read, op: pg_query(4.8fs2 epoch 5774) -2863> 2014-11-08 18:15:34.185857 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 0.000000, event: throttled, op: pg_query(4.8fs2 epoch 5774) -2862> 2014-11-08 18:15:34.185864 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 0.000000, event: all_read, op: pg_query(4.8fs2 epoch 5774) -2861> 2014-11-08 18:15:34.185872 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 2014-11-08 18:15:34.185845, event: dispatched, op: pg_query(4.8fs2 epoch 5774) -2860> 2014-11-08 18:15:34.185881 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 2014-11-08 18:15:34.185881, event: waiting_for_osdmap, op: pg_query(4.8fs2 epoch 5774) -2859> 2014-11-08 18:15:34.185891 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 2014-11-08 18:15:34.185891, event: started, op: pg_query(4.8fs2 epoch 5774) -2858> 2014-11-08 18:15:34.185907 7fc5b9b29700 5 -- op tracker -- seq: 47237, time: 2014-11-08 18:15:34.185907, event: done, op: pg_query(4.8fs2 epoch 5774) -2857> 2014-11-08 18:15:34.186024 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=2936-5750/60 luod=0'0 crt=5734'16540 active] exit Started/ReplicaActive/RepNotRecovering 1665.094310 65 0.000312 -2856> 2014-11-08 18:15:34.186045 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=2936-5750/60 luod=0'0 crt=5734'16540 active] exit Started/ReplicaActive 1665.094341 0 0.000000 -2855> 2014-11-08 18:15:34.186061 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=2936-5750/60 luod=0'0 crt=5734'16540 active] exit Started 1666.288457 0 0.000000 -2854> 2014-11-08 18:15:34.186075 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=2936-5750/60 luod=0'0 crt=5734'16540 active] enter Reset -2853> 2014-11-08 18:15:34.186184 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] exit Reset 0.000109 1 0.000084 -2852> 2014-11-08 18:15:34.186200 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] enter Started -2851> 2014-11-08 18:15:34.186214 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] enter Start -2850> 2014-11-08 18:15:34.186226 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] state<Start>: transitioning to Stray -2849> 2014-11-08 18:15:34.186240 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] exit Start 0.000027 0 0.000000 -2848> 2014-11-08 18:15:34.186254 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] enter Started/Stray -2847> 2014-11-08 18:15:34.186896 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'16633 active] exit Started/ReplicaActive/RepNotRecovering 1665.855479 65 0.000363 -2846> 2014-11-08 18:15:34.186917 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'16633 active] exit Started/ReplicaActive 1665.855509 0 0.000000 -2845> 2014-11-08 18:15:34.186932 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'16633 active] exit Started 1666.289048 0 0.000000 -2844> 2014-11-08 18:15:34.186947 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5751/5751/5639) [2,9,4] r=1 lpr=5751 pi=5635-5750/10 luod=0'0 crt=5734'16633 active] enter Reset -2843> 2014-11-08 18:15:34.186986 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'16633 inactive NOTIFY] exit Reset 0.000040 1 0.000084 -2842> 2014-11-08 18:15:34.187002 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'16633 inactive NOTIFY] enter Started -2841> 2014-11-08 18:15:34.187015 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'16633 inactive NOTIFY] enter Start -2840> 2014-11-08 18:15:34.187028 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'16633 inactive NOTIFY] state<Start>: transitioning to Stray -2839> 2014-11-08 18:15:34.187050 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'16633 inactive NOTIFY] exit Start 0.000035 0 0.000000 -2838> 2014-11-08 18:15:34.187064 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5bs1( v 5734'16639 (3473'13638,5734'16639] local-les=5752 n=15505 ec=195 les/c 5752/5755 5774/5774/5639) [2,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'16633 inactive NOTIFY] enter Started/Stray -2837> 2014-11-08 18:15:34.187135 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8646 ==== pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) v5 ==== 8766+0+0 (3099820115 0 0) 0x2234a760 con 0x1c5c0580 -2836> 2014-11-08 18:15:34.187151 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.186843, event: header_read, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2835> 2014-11-08 18:15:34.187162 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.186844, event: throttled, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2834> 2014-11-08 18:15:34.187171 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.187119, event: all_read, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2833> 2014-11-08 18:15:34.187181 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.187150, event: dispatched, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2832> 2014-11-08 18:15:34.187190 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.187190, event: waiting_for_osdmap, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2831> 2014-11-08 18:15:34.187201 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.187201, event: started, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2830> 2014-11-08 18:15:34.187238 7fc5b9b29700 5 -- op tracker -- seq: 47238, time: 2014-11-08 18:15:34.187238, event: done, op: pg_notify(4.93s1(7),4.98s2(24),4.9ds2(46),4.a0s2(21) epoch 5774) -2829> 2014-11-08 18:15:34.187911 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5751/5751/5641) [1,4,9]/[1,2147483647,9] r=2 lpr=5751 pi=2790-5750/92 luod=0'0 crt=5734'16383 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1661.177791 57 0.000299 -2828> 2014-11-08 18:15:34.187943 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5751/5751/5641) [1,4,9]/[1,2147483647,9] r=2 lpr=5751 pi=2790-5750/92 luod=0'0 crt=5734'16383 active+remapped] exit Started/ReplicaActive 1661.177837 0 0.000000 -2827> 2014-11-08 18:15:34.187960 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5751/5751/5641) [1,4,9]/[1,2147483647,9] r=2 lpr=5751 pi=2790-5750/92 luod=0'0 crt=5734'16383 active+remapped] exit Started 1666.289635 0 0.000000 -2826> 2014-11-08 18:15:34.187976 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5751/5751/5641) [1,4,9]/[1,2147483647,9] r=2 lpr=5751 pi=2790-5750/92 luod=0'0 crt=5734'16383 active+remapped] enter Reset -2825> 2014-11-08 18:15:34.188088 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] exit Reset 0.000113 1 0.000098 -2824> 2014-11-08 18:15:34.188104 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] enter Started -2823> 2014-11-08 18:15:34.188117 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] enter Start -2822> 2014-11-08 18:15:34.188129 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] state<Start>: transitioning to Stray -2821> 2014-11-08 18:15:34.188144 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] exit Start 0.000027 0 0.000000 -2820> 2014-11-08 18:15:34.188158 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] enter Started/Stray -2819> 2014-11-08 18:15:34.188780 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5751/5751/5751) [9,4,1]/[2147483647,4,10] r=-1 lpr=5751 pi=2322-5750/90 luod=0'0 crt=5734'16448 active+remapped] exit Started/ReplicaActive/RepRecovering 1025.983655 49 0.000095 -2818> 2014-11-08 18:15:34.188804 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5751/5751/5751) [9,4,1]/[2147483647,4,10] r=-1 lpr=5751 pi=2322-5750/90 luod=0'0 crt=5734'16448 active+remapped] exit Started/ReplicaActive 1664.843055 0 0.000000 -2817> 2014-11-08 18:15:34.188820 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5751/5751/5751) [9,4,1]/[2147483647,4,10] r=-1 lpr=5751 pi=2322-5750/90 luod=0'0 crt=5734'16448 active+remapped] exit Started 1666.290006 0 0.000000 -2816> 2014-11-08 18:15:34.188836 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5751/5751/5751) [9,4,1]/[2147483647,4,10] r=-1 lpr=5751 pi=2322-5750/90 luod=0'0 crt=5734'16448 active+remapped] enter Reset -2815> 2014-11-08 18:15:34.188906 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5774/5774/5774) [9,2147483647,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=2322-5773/91 crt=5734'16448 remapped NOTIFY] exit Reset 0.000070 1 0.000089 -2814> 2014-11-08 18:15:34.188923 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5774/5774/5774) [9,2147483647,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=2322-5773/91 crt=5734'16448 remapped NOTIFY] enter Started -2813> 2014-11-08 18:15:34.188937 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5774/5774/5774) [9,2147483647,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=2322-5773/91 crt=5734'16448 remapped NOTIFY] enter Start -2812> 2014-11-08 18:15:34.188951 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5774/5774/5774) [9,2147483647,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=2322-5773/91 crt=5734'16448 remapped NOTIFY] state<Start>: transitioning to Stray -2811> 2014-11-08 18:15:34.188966 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5774/5774/5774) [9,2147483647,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=2322-5773/91 crt=5734'16448 remapped NOTIFY] exit Start 0.000029 0 0.000000 -2810> 2014-11-08 18:15:34.188982 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5ds0( v 5734'16451 (3473'13379,5734'16451] lb 629bbddd/1000001a794.000001d6/head//4 local-les=0 n=15809 ec=195 les/c 5752/4877 5774/5774/5774) [9,2147483647,1]/[2147483647,2147483647,10] r=-1 lpr=5774 pi=2322-5773/91 crt=5734'16448 remapped NOTIFY] enter Started/Stray -2809> 2014-11-08 18:15:34.189198 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5751/5751/5641) [1,9,4]/[1,9,2147483647] r=1 lpr=5751 pi=4252-5750/68 luod=0'0 crt=5734'16425 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1661.178790 57 0.000342 -2808> 2014-11-08 18:15:34.189218 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5751/5751/5641) [1,9,4]/[1,9,2147483647] r=1 lpr=5751 pi=4252-5750/68 luod=0'0 crt=5734'16425 active+remapped] exit Started/ReplicaActive 1661.178818 0 0.000000 -2807> 2014-11-08 18:15:34.189233 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5751/5751/5641) [1,9,4]/[1,9,2147483647] r=1 lpr=5751 pi=4252-5750/68 luod=0'0 crt=5734'16425 active+remapped] exit Started 1666.277344 0 0.000000 -2806> 2014-11-08 18:15:34.189249 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5751/5751/5641) [1,9,4]/[1,9,2147483647] r=1 lpr=5751 pi=4252-5750/68 luod=0'0 crt=5734'16425 active+remapped] enter Reset -2805> 2014-11-08 18:15:34.189791 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] exit Reset 0.000543 1 0.000080 -2804> 2014-11-08 18:15:34.189805 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] enter Started -2803> 2014-11-08 18:15:34.189813 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] enter Start -2802> 2014-11-08 18:15:34.189821 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] state<Start>: transitioning to Stray -2801> 2014-11-08 18:15:34.189830 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] exit Start 0.000017 0 0.000000 -2800> 2014-11-08 18:15:34.189839 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] enter Started/Stray -2799> 2014-11-08 18:15:34.189949 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5759/5639) [2,5,9]/[2,2147483647,4] r=-1 lpr=5759 pi=4863-5758/53 luod=0'0 crt=5734'16346 active+remapped] exit Started/ReplicaActive/RepNotRecovering 965.528963 27 0.000248 -2798> 2014-11-08 18:15:34.189963 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5759/5639) [2,5,9]/[2,2147483647,4] r=-1 lpr=5759 pi=4863-5758/53 luod=0'0 crt=5734'16346 active+remapped] exit Started/ReplicaActive 965.528988 0 0.000000 -2797> 2014-11-08 18:15:34.189976 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5759/5639) [2,5,9]/[2,2147483647,4] r=-1 lpr=5759 pi=4863-5758/53 luod=0'0 crt=5734'16346 active+remapped] exit Started 1015.782475 0 0.000000 -2796> 2014-11-08 18:15:34.189997 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5759/5639) [2,5,9]/[2,2147483647,4] r=-1 lpr=5759 pi=4863-5758/53 luod=0'0 crt=5734'16346 active+remapped] enter Reset -2795> 2014-11-08 18:15:34.190121 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12088 ==== pg_notify(4.8fs1(21) epoch 5774) v5 ==== 2014+0+0 (1380983448 0 0) 0x2234a1c0 con 0x1c5c2680 -2794> 2014-11-08 18:15:34.190133 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190042, event: header_read, op: pg_notify(4.8fs1(21) epoch 5774) -2793> 2014-11-08 18:15:34.190139 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190043, event: throttled, op: pg_notify(4.8fs1(21) epoch 5774) -2792> 2014-11-08 18:15:34.190142 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190105, event: all_read, op: pg_notify(4.8fs1(21) epoch 5774) -2791> 2014-11-08 18:15:34.190146 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190131, event: dispatched, op: pg_notify(4.8fs1(21) epoch 5774) -2790> 2014-11-08 18:15:34.190149 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190149, event: waiting_for_osdmap, op: pg_notify(4.8fs1(21) epoch 5774) -2789> 2014-11-08 18:15:34.190155 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190155, event: started, op: pg_notify(4.8fs1(21) epoch 5774) -2788> 2014-11-08 18:15:34.190171 7fc5b9b29700 5 -- op tracker -- seq: 47239, time: 2014-11-08 18:15:34.190170, event: done, op: pg_notify(4.8fs1(21) epoch 5774) -2787> 2014-11-08 18:15:34.190828 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5774/5639) [2,5,9]/[2,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/54 crt=5734'16346 remapped NOTIFY] exit Reset 0.000832 1 0.000835 -2786> 2014-11-08 18:15:34.190843 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5774/5639) [2,5,9]/[2,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/54 crt=5734'16346 remapped NOTIFY] enter Started -2785> 2014-11-08 18:15:34.190854 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5774/5639) [2,5,9]/[2,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/54 crt=5734'16346 remapped NOTIFY] enter Start -2784> 2014-11-08 18:15:34.190863 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5774/5639) [2,5,9]/[2,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/54 crt=5734'16346 remapped NOTIFY] state<Start>: transitioning to Stray -2783> 2014-11-08 18:15:34.190872 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5774/5639) [2,5,9]/[2,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/54 crt=5734'16346 remapped NOTIFY] exit Start 0.000019 0 0.000000 -2782> 2014-11-08 18:15:34.190884 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.49s2( v 5734'16346 (3473'13272,5734'16346] lb 108dd049/100000294fd.00000021/head//4 local-les=0 n=408 ec=195 les/c 5765/4864 5759/5774/5639) [2,5,9]/[2,2147483647,2147483647] r=-1 lpr=5774 pi=4863-5773/54 crt=5734'16346 remapped NOTIFY] enter Started/Stray -2781> 2014-11-08 18:15:34.191000 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5751/5751/5751) [4,9,1]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'16569 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1664.968095 64 0.000301 -2780> 2014-11-08 18:15:34.191017 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5751/5751/5751) [4,9,1]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'16569 active+remapped] exit Started/ReplicaActive 1664.968122 0 0.000000 -2779> 2014-11-08 18:15:34.191027 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5751/5751/5751) [4,9,1]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'16569 active+remapped] exit Started 1666.271596 0 0.000000 -2778> 2014-11-08 18:15:34.191037 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5751/5751/5751) [4,9,1]/[4,9,2147483647] r=1 lpr=5751 pi=4266-5750/69 luod=0'0 crt=5734'16569 active+remapped] enter Reset -2777> 2014-11-08 18:15:34.191070 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] exit Reset 0.000033 1 0.000069 -2776> 2014-11-08 18:15:34.191079 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] enter Started -2775> 2014-11-08 18:15:34.191086 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] enter Start -2774> 2014-11-08 18:15:34.191092 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] state<Start>: transitioning to Primary -2773> 2014-11-08 18:15:34.191100 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -2772> 2014-11-08 18:15:34.191108 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] enter Started/Primary -2771> 2014-11-08 18:15:34.191115 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped] enter Started/Primary/Peering -2770> 2014-11-08 18:15:34.191124 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4es1( v 5734'16576 (3473'13515,5734'16576] local-les=5752 n=15408 ec=195 les/c 5752/5636 5774/5774/5774) [2147483647,9,1]/[2147483647,9,2147483647] r=1 lpr=5774 pi=4266-5773/70 crt=5734'16569 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -2769> 2014-11-08 18:15:34.191251 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5751/5768/5768) [9,1,4] r=0 lpr=5768 crt=5734'16340 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 710.843939 13 0.000101 -2768> 2014-11-08 18:15:34.191267 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5751/5768/5768) [9,1,4] r=0 lpr=5768 crt=5734'16340 mlcod 0'0 active] exit Started/Primary/Active 711.374935 0 0.000000 -2767> 2014-11-08 18:15:34.191276 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5751/5768/5768) [9,1,4] r=0 lpr=5768 crt=5734'16340 mlcod 0'0 active] exit Started/Primary 720.279658 0 0.000000 -2766> 2014-11-08 18:15:34.191285 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5751/5768/5768) [9,1,4] r=0 lpr=5768 crt=5734'16340 mlcod 0'0 active] exit Started 720.279693 0 0.000000 -2765> 2014-11-08 18:15:34.191294 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5751/5768/5768) [9,1,4] r=0 lpr=5768 crt=5734'16340 mlcod 0'0 active] enter Reset -2764> 2014-11-08 18:15:34.191336 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] exit Reset 0.000042 1 0.000085 -2763> 2014-11-08 18:15:34.191345 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] enter Started -2762> 2014-11-08 18:15:34.191351 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] enter Start -2761> 2014-11-08 18:15:34.191357 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2760> 2014-11-08 18:15:34.191364 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] exit Start 0.000013 0 0.000000 -2759> 2014-11-08 18:15:34.191371 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] enter Started/Primary -2758> 2014-11-08 18:15:34.191377 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] enter Started/Primary/Peering -2757> 2014-11-08 18:15:34.191383 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2756> 2014-11-08 18:15:34.191434 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16225 active] exit Started/ReplicaActive/RepNotRecovering 1664.968220 65 0.000373 -2755> 2014-11-08 18:15:34.191444 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16225 active] exit Started/ReplicaActive 1664.968238 0 0.000000 -2754> 2014-11-08 18:15:34.191452 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16225 active] exit Started 1666.270095 0 0.000000 -2753> 2014-11-08 18:15:34.191459 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5751/5751/5751) [4,1,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16225 active] enter Reset -2752> 2014-11-08 18:15:34.191510 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] exit Reset 0.000051 1 0.000047 -2751> 2014-11-08 18:15:34.191519 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] enter Started -2750> 2014-11-08 18:15:34.191525 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] enter Start -2749> 2014-11-08 18:15:34.191530 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] state<Start>: transitioning to Stray -2748> 2014-11-08 18:15:34.191537 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] exit Start 0.000012 0 0.000000 -2747> 2014-11-08 18:15:34.191544 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] enter Started/Stray -2746> 2014-11-08 18:15:34.191603 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1651.081497 52 0.000191 -2745> 2014-11-08 18:15:34.191613 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active] exit Started/Primary/Active 1664.973335 0 0.000000 -2744> 2014-11-08 18:15:34.191621 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active] exit Started/Primary 1666.262706 0 0.000000 -2743> 2014-11-08 18:15:34.191628 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active] exit Started 1666.262772 0 0.000000 -2742> 2014-11-08 18:15:34.191634 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active] enter Reset -2741> 2014-11-08 18:15:34.191661 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] exit Reset 0.000027 1 0.000058 -2740> 2014-11-08 18:15:34.191670 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] enter Started -2739> 2014-11-08 18:15:34.191676 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] enter Start -2738> 2014-11-08 18:15:34.191682 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2737> 2014-11-08 18:15:34.191861 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] exit Start 0.000185 0 0.000000 -2736> 2014-11-08 18:15:34.191872 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] enter Started/Primary -2735> 2014-11-08 18:15:34.191880 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] enter Started/Primary/Peering -2734> 2014-11-08 18:15:34.191888 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2733> 2014-11-08 18:15:34.191939 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5759/5728) [9,2,5]/[9,4,2147483647] r=0 lpr=5759 pi=4872-5758/52 bft=2(1),5(2) crt=5734'8087 lcod 5734'8093 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 961.093607 25 0.000143 -2732> 2014-11-08 18:15:34.191955 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5759/5728) [9,2,5]/[9,4,2147483647] r=0 lpr=5759 pi=4872-5758/52 bft=2(1),5(2) crt=5734'8087 lcod 5734'8093 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 965.471419 0 0.000000 -2731> 2014-11-08 18:15:34.191969 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5759/5728) [9,2,5]/[9,4,2147483647] r=0 lpr=5759 pi=4872-5758/52 bft=2(1),5(2) crt=5734'8087 lcod 5734'8093 mlcod 0'0 active+remapped] exit Started/Primary 1015.530640 0 0.000000 -2730> 2014-11-08 18:15:34.191980 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5759/5728) [9,2,5]/[9,4,2147483647] r=0 lpr=5759 pi=4872-5758/52 bft=2(1),5(2) crt=5734'8087 lcod 5734'8093 mlcod 0'0 active+remapped] exit Started 1015.530682 0 0.000000 -2729> 2014-11-08 18:15:34.191991 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5759/5728) [9,2,5]/[9,4,2147483647] r=0 lpr=5759 pi=4872-5758/52 bft=2(1),5(2) crt=5734'8087 lcod 5734'8093 mlcod 0'0 active+remapped] enter Reset -2728> 2014-11-08 18:15:34.192024 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] exit Reset 0.000033 1 0.000084 -2727> 2014-11-08 18:15:34.192035 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] enter Started -2726> 2014-11-08 18:15:34.192044 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] enter Start -2725> 2014-11-08 18:15:34.192052 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] state<Start>: transitioning to Primary -2724> 2014-11-08 18:15:34.192062 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] exit Start 0.000018 0 0.000000 -2723> 2014-11-08 18:15:34.192072 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] enter Started/Primary -2722> 2014-11-08 18:15:34.192080 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped] enter Started/Primary/Peering -2721> 2014-11-08 18:15:34.192089 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.38s0( v 5734'8094 (3159'5067,5734'8094] local-les=5765 n=7505 ec=195 les/c 5765/4877 5759/5774/5728) [9,2,5]/[9,2147483647,2147483647] r=0 lpr=5774 pi=4872-5773/53 crt=5734'8087 lcod 5734'8093 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -2720> 2014-11-08 18:15:34.192161 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.59s2(61),4.5as2(93),4.57s1(69),4.45s2(34) epoch 5774) v5 -- ?+0 0x2234b660 con 0x1c5c2680 -2719> 2014-11-08 18:15:34.192175 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.5bs1(11),4.49s2(54) epoch 5774) v5 -- ?+0 0x2234ba20 con 0x1c5c0580 -2718> 2014-11-08 18:15:34.192305 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.5ds0(91) epoch 5774) v5 -- ?+0 0x2234c380 con 0x1c7911e0 -2717> 2014-11-08 18:15:34.192447 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_query(4.39s1,4.40s1,4.4es2 epoch 5774) v3 -- ?+0 0x2234c740 con 0x1c5c2680 -2716> 2014-11-08 18:15:34.192457 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_query(4.38s1 epoch 5774) v3 -- ?+0 0x25f930c0 con 0x1c5c0580 -2715> 2014-11-08 18:15:34.192501 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_query(4.38s2 epoch 5774) v3 -- ?+0 0x1d0694a0 con 0x1e50aec0 -2714> 2014-11-08 18:15:34.192583 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5759/5751) [5,2,10]/[2147483647,4,2] r=-1 lpr=5759 pi=5453-5758/37 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started/Stray 1015.530803 45 0.000913 -2713> 2014-11-08 18:15:34.192598 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5759/5751) [5,2,10]/[2147483647,4,2] r=-1 lpr=5759 pi=5453-5758/37 crt=0'0 lcod 0'0 remapped NOTIFY] exit Started 1015.530847 0 0.000000 -2712> 2014-11-08 18:15:34.192606 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5759/5751) [5,2,10]/[2147483647,4,2] r=-1 lpr=5759 pi=5453-5758/37 crt=0'0 lcod 0'0 remapped NOTIFY] enter Reset -2711> 2014-11-08 18:15:34.192670 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5774/5774) [5,2,10]/[2147483647,2147483647,2] r=-1 lpr=5774 pi=5453-5773/38 crt=0'0 lcod 0'0 remapped NOTIFY] exit Reset 0.000063 1 0.000047 -2710> 2014-11-08 18:15:34.192680 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5774/5774) [5,2,10]/[2147483647,2147483647,2] r=-1 lpr=5774 pi=5453-5773/38 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started -2709> 2014-11-08 18:15:34.192694 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5774/5774) [5,2,10]/[2147483647,2147483647,2] r=-1 lpr=5774 pi=5453-5773/38 crt=0'0 lcod 0'0 remapped NOTIFY] enter Start -2708> 2014-11-08 18:15:34.192752 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5774/5774) [5,2,10]/[2147483647,2147483647,2] r=-1 lpr=5774 pi=5453-5773/38 crt=0'0 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray -2707> 2014-11-08 18:15:34.192762 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5774/5774) [5,2,10]/[2147483647,2147483647,2] r=-1 lpr=5774 pi=5453-5773/38 crt=0'0 lcod 0'0 remapped NOTIFY] exit Start 0.000069 0 0.000000 -2706> 2014-11-08 18:15:34.192770 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.3fs0( v 5390'8059 (3458'5059,5390'8059] lb 0//0//-1 local-les=0 n=0 ec=195 les/c 5464/5454 5759/5774/5774) [5,2,10]/[2147483647,2147483647,2] r=-1 lpr=5774 pi=5453-5773/38 crt=0'0 lcod 0'0 remapped NOTIFY] enter Started/Stray -2705> 2014-11-08 18:15:34.192911 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2332-5750/43 luod=0'0 crt=5734'8187 active] exit Started/ReplicaActive/RepNotRecovering 1665.866706 65 0.000488 -2704> 2014-11-08 18:15:34.192922 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2332-5750/43 luod=0'0 crt=5734'8187 active] exit Started/ReplicaActive 1665.866728 0 0.000000 -2703> 2014-11-08 18:15:34.192931 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2332-5750/43 luod=0'0 crt=5734'8187 active] exit Started 1666.294077 0 0.000000 -2702> 2014-11-08 18:15:34.192938 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2332-5750/43 luod=0'0 crt=5734'8187 active] enter Reset -2701> 2014-11-08 18:15:34.193036 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2332-5773/44 crt=5734'8187 inactive NOTIFY] exit Reset 0.000097 1 0.000050 -2700> 2014-11-08 18:15:34.193047 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2332-5773/44 crt=5734'8187 inactive NOTIFY] enter Started -2699> 2014-11-08 18:15:34.193056 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2332-5773/44 crt=5734'8187 inactive NOTIFY] enter Start -2698> 2014-11-08 18:15:34.193063 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2332-5773/44 crt=5734'8187 inactive NOTIFY] state<Start>: transitioning to Stray -2697> 2014-11-08 18:15:34.193072 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2332-5773/44 crt=5734'8187 inactive NOTIFY] exit Start 0.000016 0 0.000000 -2696> 2014-11-08 18:15:34.193080 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.33s2( v 5734'8192 (3458'5191,5734'8192] local-les=5752 n=7630 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2332-5773/44 crt=5734'8187 inactive NOTIFY] enter Started/Stray -2695> 2014-11-08 18:15:34.193214 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1651.095303 52 0.003182 -2694> 2014-11-08 18:15:34.193250 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active] exit Started/Primary/Active 1665.042085 0 0.000000 -2693> 2014-11-08 18:15:34.193259 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active] exit Started/Primary 1666.284860 0 0.000000 -2692> 2014-11-08 18:15:34.193266 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active] exit Started 1666.284905 0 0.000000 -2691> 2014-11-08 18:15:34.193282 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5751/5751/5729) [9,4,1] r=0 lpr=5751 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active] enter Reset -2690> 2014-11-08 18:15:34.193328 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8647 ==== pg_notify(4.38s1(53) epoch 5774) v5 ==== 3902+0+0 (1139384728 0 0) 0x25f930c0 con 0x1c5c0580 -2689> 2014-11-08 18:15:34.193339 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193258, event: header_read, op: pg_notify(4.38s1(53) epoch 5774) -2688> 2014-11-08 18:15:34.193344 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193259, event: throttled, op: pg_notify(4.38s1(53) epoch 5774) -2687> 2014-11-08 18:15:34.193346 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193317, event: all_read, op: pg_notify(4.38s1(53) epoch 5774) -2686> 2014-11-08 18:15:34.193348 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193337, event: dispatched, op: pg_notify(4.38s1(53) epoch 5774) -2685> 2014-11-08 18:15:34.193351 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193351, event: waiting_for_osdmap, op: pg_notify(4.38s1(53) epoch 5774) -2684> 2014-11-08 18:15:34.193355 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193355, event: started, op: pg_notify(4.38s1(53) epoch 5774) -2683> 2014-11-08 18:15:34.193367 7fc5b9b29700 5 -- op tracker -- seq: 47240, time: 2014-11-08 18:15:34.193367, event: done, op: pg_notify(4.38s1(53) epoch 5774) -2682> 2014-11-08 18:15:34.193313 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] exit Reset 0.000031 1 0.000099 -2681> 2014-11-08 18:15:34.193402 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] enter Started -2680> 2014-11-08 18:15:34.193415 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] enter Start -2679> 2014-11-08 18:15:34.193428 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2678> 2014-11-08 18:15:34.193441 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] exit Start 0.000026 0 0.000000 -2677> 2014-11-08 18:15:34.193451 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] enter Started/Primary -2676> 2014-11-08 18:15:34.193459 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] enter Started/Primary/Peering -2675> 2014-11-08 18:15:34.193468 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2674> 2014-11-08 18:15:34.193550 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5751/5759/5728) [1,4,10]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/39 bft=1(0),4(1),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 959.632412 33 0.000148 -2673> 2014-11-08 18:15:34.193564 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5751/5759/5728) [1,4,10]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/39 bft=1(0),4(1),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active 965.474029 0 0.000000 -2672> 2014-11-08 18:15:34.193576 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5751/5759/5728) [1,4,10]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/39 bft=1(0),4(1),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 active+remapped] exit Started/Primary 1015.531480 0 0.000000 -2671> 2014-11-08 18:15:34.193585 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5751/5759/5728) [1,4,10]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/39 bft=1(0),4(1),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 active+remapped] exit Started 1015.531521 0 0.000000 -2670> 2014-11-08 18:15:34.193593 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5751/5759/5728) [1,4,10]/[2147483647,9,5] r=1 lpr=5759 pi=5453-5758/39 bft=1(0),4(1),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 active+remapped] enter Reset -2669> 2014-11-08 18:15:34.193662 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] exit Reset 0.000069 1 0.000077 -2668> 2014-11-08 18:15:34.193678 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] enter Started -2667> 2014-11-08 18:15:34.193693 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] enter Start -2666> 2014-11-08 18:15:34.193700 7fc5b1b19700 1 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] state<Start>: transitioning to Primary -2665> 2014-11-08 18:15:34.193708 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] exit Start 0.000014 0 0.000000 -2664> 2014-11-08 18:15:34.193715 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] enter Started/Primary -2663> 2014-11-08 18:15:34.193722 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] enter Started/Primary/Peering -2662> 2014-11-08 18:15:34.193729 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetInfo -2661> 2014-11-08 18:15:34.193864 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.028491 2 0.000066 -2660> 2014-11-08 18:15:34.193881 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2659> 2014-11-08 18:15:34.193904 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000023 0 0.000000 -2658> 2014-11-08 18:15:34.193915 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2657> 2014-11-08 18:15:34.193925 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000011 0 0.000000 -2656> 2014-11-08 18:15:34.193934 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2655> 2014-11-08 18:15:34.193955 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.028001 2 0.000066 -2654> 2014-11-08 18:15:34.193965 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2653> 2014-11-08 18:15:34.193980 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000015 0 0.000000 -2652> 2014-11-08 18:15:34.193990 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2651> 2014-11-08 18:15:34.193998 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000008 0 0.000000 -2650> 2014-11-08 18:15:34.194006 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2649> 2014-11-08 18:15:34.194025 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.028488 2 0.000062 -2648> 2014-11-08 18:15:34.194035 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2647> 2014-11-08 18:15:34.194059 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000024 0 0.000000 -2646> 2014-11-08 18:15:34.194071 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2645> 2014-11-08 18:15:34.194079 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000008 0 0.000000 -2644> 2014-11-08 18:15:34.194087 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2643> 2014-11-08 18:15:34.194188 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.027242 2 0.016017 -2642> 2014-11-08 18:15:34.194200 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2641> 2014-11-08 18:15:34.194218 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2640> 2014-11-08 18:15:34.194230 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2639> 2014-11-08 18:15:34.194240 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2638> 2014-11-08 18:15:34.194250 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2637> 2014-11-08 18:15:34.194287 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.027951 2 0.000060 -2636> 2014-11-08 18:15:34.194299 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2635> 2014-11-08 18:15:34.194316 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2634> 2014-11-08 18:15:34.194328 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2633> 2014-11-08 18:15:34.194337 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2632> 2014-11-08 18:15:34.194347 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2631> 2014-11-08 18:15:34.194376 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.024933 2 0.000072 -2630> 2014-11-08 18:15:34.194387 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2629> 2014-11-08 18:15:34.194405 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2628> 2014-11-08 18:15:34.194416 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2627> 2014-11-08 18:15:34.194426 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2626> 2014-11-08 18:15:34.194436 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2625> 2014-11-08 18:15:34.194456 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.ds1(11),4.12s1(53),4.24s1(11) epoch 5774) v5 -- ?+0 0x22a26180 con 0x1c5c2680 -2624> 2014-11-08 18:15:34.194469 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.3fs0(38),4.33s2(44) epoch 5774) v5 -- ?+0 0x22a21860 con 0x1c5c0580 -2623> 2014-11-08 18:15:34.194600 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_query(4.31s2,4.36s0 epoch 5774) v3 -- ?+0 0x1dbfb2a0 con 0x1c5c2680 -2622> 2014-11-08 18:15:34.194611 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_query(4.36s2 epoch 5774) v3 -- ?+0 0x1dbf9c20 con 0x1e50aec0 -2621> 2014-11-08 18:15:34.194665 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_query(4.36s2 epoch 5774) v3 -- ?+0 0x21221860 con 0x1c7911e0 -2620> 2014-11-08 18:15:34.195054 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.024003 2 0.000167 -2619> 2014-11-08 18:15:34.195066 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2618> 2014-11-08 18:15:34.195084 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2617> 2014-11-08 18:15:34.195097 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2616> 2014-11-08 18:15:34.195108 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2615> 2014-11-08 18:15:34.195118 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2614> 2014-11-08 18:15:34.195154 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.023582 2 0.000290 -2613> 2014-11-08 18:15:34.195166 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2612> 2014-11-08 18:15:34.195184 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2611> 2014-11-08 18:15:34.195196 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2610> 2014-11-08 18:15:34.195206 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2609> 2014-11-08 18:15:34.195216 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2608> 2014-11-08 18:15:34.195249 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.014716 2 0.000126 -2607> 2014-11-08 18:15:34.195261 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2606> 2014-11-08 18:15:34.195280 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2605> 2014-11-08 18:15:34.195291 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2604> 2014-11-08 18:15:34.195301 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000009 0 0.000000 -2603> 2014-11-08 18:15:34.195311 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2602> 2014-11-08 18:15:34.195333 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.013885 2 0.000652 -2601> 2014-11-08 18:15:34.195345 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2600> 2014-11-08 18:15:34.195363 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2599> 2014-11-08 18:15:34.195374 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2598> 2014-11-08 18:15:34.195384 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2597> 2014-11-08 18:15:34.195394 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2596> 2014-11-08 18:15:34.195639 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.030573 2 0.014093 -2595> 2014-11-08 18:15:34.195650 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2594> 2014-11-08 18:15:34.195667 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000017 0 0.000000 -2593> 2014-11-08 18:15:34.195676 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2592> 2014-11-08 18:15:34.195692 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000016 0 0.000000 -2591> 2014-11-08 18:15:34.195701 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2590> 2014-11-08 18:15:34.195721 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MBackfillReserve REQUEST pgid: 4.20s2, query_epoch: 5774, prio: 201 v3 -- ?+0 0x213beae0 con 0x1e50aec0 -2589> 2014-11-08 18:15:34.195782 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.7s2(24),4.99s1(11),4.9fs2(42),4.59s2(61),4.5as2(93) epoch 5774) v5 -- ?+0 0x2286d820 con 0x1c5c2680 -2588> 2014-11-08 18:15:34.195793 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.6s2(24),4.b2s2(71),4.b5s1(22),4.bcs1(76),4.bds1(19),4.94s1(11),4.9cs2(22) epoch 5774) v5 -- ?+0 0x2127a760 con 0x1c5c0580 -2587> 2014-11-08 18:15:34.196117 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.88s1( v 5734'8208 (3458'5184,5734'8208] lb 5661fd88/1000002b663.0000000d/head//4 local-les=0 n=6651 ec=195 les/c 5761/5721 5759/5759/5641) [1,9,5]/[1,2147483647,5] r=-1 lpr=5759 pi=4863-5758/39 luod=0'0 crt=5734'8199 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1012.269210 42 0.000275 -2586> 2014-11-08 18:15:34.196131 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.88s1( v 5734'8208 (3458'5184,5734'8208] lb 5661fd88/1000002b663.0000000d/head//4 local-les=0 n=6651 ec=195 les/c 5761/5721 5759/5759/5641) [1,9,5]/[1,2147483647,5] r=-1 lpr=5759 pi=4863-5758/39 luod=0'0 crt=5734'8199 active+remapped] enter Started/ReplicaActive/RepWaitBackfillReserved -2585> 2014-11-08 18:15:34.196222 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.89s2( v 5734'8486 (3458'5440,5734'8486] lb 40520589/100000199b6.00000035/head//4 local-les=0 n=2397 ec=195 les/c 5760/5721 5759/5759/5639) [2,5,9]/[2,5,2147483647] r=-1 lpr=5759 pi=5432-5758/33 luod=0'0 crt=5734'8486 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1015.355402 45 0.000317 -2584> 2014-11-08 18:15:34.196235 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.89s2( v 5734'8486 (3458'5440,5734'8486] lb 40520589/100000199b6.00000035/head//4 local-les=0 n=2397 ec=195 les/c 5760/5721 5759/5759/5639) [2,5,9]/[2,5,2147483647] r=-1 lpr=5759 pi=5432-5758/33 luod=0'0 crt=5734'8486 active+remapped] enter Started/ReplicaActive/RepWaitBackfillReserved -2583> 2014-11-08 18:15:34.196362 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.004979 2 0.000061 -2582> 2014-11-08 18:15:34.196374 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2581> 2014-11-08 18:15:34.196394 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000020 0 0.000000 -2580> 2014-11-08 18:15:34.196406 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2579> 2014-11-08 18:15:34.196416 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2578> 2014-11-08 18:15:34.196426 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2577> 2014-11-08 18:15:34.196449 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.004561 2 0.000243 -2576> 2014-11-08 18:15:34.196460 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2575> 2014-11-08 18:15:34.196478 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000018 0 0.000000 -2574> 2014-11-08 18:15:34.196489 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2573> 2014-11-08 18:15:34.196499 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000010 0 0.000000 -2572> 2014-11-08 18:15:34.196510 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2571> 2014-11-08 18:15:34.196531 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.003064 2 0.000171 -2570> 2014-11-08 18:15:34.196543 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2569> 2014-11-08 18:15:34.196567 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000024 0 0.000000 -2568> 2014-11-08 18:15:34.196578 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2567> 2014-11-08 18:15:34.196587 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000009 0 0.000000 -2566> 2014-11-08 18:15:34.196595 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2565> 2014-11-08 18:15:34.196827 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.35s1( v 5734'8216 (3458'5173,5734'8216] local-les=5761 n=7732 ec=195 les/c 5761/5636 5759/5759/5728) [1,9,5]/[2147483647,9,5] r=1 lpr=5759 pi=5635-5758/18 bft=1(0) crt=5734'8206 lcod 5734'8215 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 1008.826582 34 0.000117 -2564> 2014-11-08 18:15:34.196840 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.35s1( v 5734'8216 (3458'5173,5734'8216] local-les=5761 n=7732 ec=195 les/c 5761/5636 5759/5759/5728) [1,9,5]/[2147483647,9,5] r=1 lpr=5759 pi=5635-5758/18 bft=1(0) crt=5734'8206 lcod 5734'8215 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] enter Started/Primary/Active/WaitRemoteBackfillReserved -2563> 2014-11-08 18:15:34.196850 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MBackfillReserve REQUEST pgid: 4.35s0, query_epoch: 5774, prio: 201 v3 -- ?+0 0x1db1f260 con 0x1c5c2680 -2562> 2014-11-08 18:15:34.196866 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4bs1( v 5734'16442 (3473'13377,5734'16442] local-les=5763 n=15306 ec=195 les/c 5763/5636 5759/5759/5728) [2,9,5]/[2147483647,9,5] r=1 lpr=5759 pi=5635-5758/18 bft=2(0) crt=5734'16437 lcod 5734'16441 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitLocalBackfillReserved 1008.798120 45 0.000151 -2561> 2014-11-08 18:15:34.196876 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4bs1( v 5734'16442 (3473'13377,5734'16442] local-les=5763 n=15306 ec=195 les/c 5763/5636 5759/5759/5728) [2,9,5]/[2147483647,9,5] r=1 lpr=5759 pi=5635-5758/18 bft=2(0) crt=5734'16437 lcod 5734'16441 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] enter Started/Primary/Active/WaitRemoteBackfillReserved -2560> 2014-11-08 18:15:34.196897 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MBackfillReserve REQUEST pgid: 4.4bs0, query_epoch: 5774, prio: 201 v3 -- ?+0 0x227cecc0 con 0x1c5c0580 -2559> 2014-11-08 18:15:34.196946 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.45s2(34),4.57s1(69) epoch 5774) v5 -- ?+0 0x227cb0c0 con 0x1c5c2680 -2558> 2014-11-08 18:15:34.196954 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.14s2(44),4.2ds2(46),4.2as2(35),4.33s2(44),4.49s2(54),4.5bs1(11) epoch 5774) v5 -- ?+0 0x227c8f00 con 0x1c5c0580 -2557> 2014-11-08 18:15:34.197215 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 0.028598 4 0.000081 -2556> 2014-11-08 18:15:34.197235 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -2555> 2014-11-08 18:15:34.197264 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000029 0 0.000000 -2554> 2014-11-08 18:15:34.197279 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -2553> 2014-11-08 18:15:34.197291 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000012 0 0.000000 -2552> 2014-11-08 18:15:34.197302 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -2551> 2014-11-08 18:15:34.197478 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.21s2(50) epoch 5774) v5 -- ?+0 0x227cbc00 con 0x42c86e0 -2550> 2014-11-08 18:15:34.197497 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.8fs2(21) epoch 5774) v5 -- ?+0 0x227ca1c0 con 0x42c86e0 -2549> 2014-11-08 18:15:34.197508 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.21s2(50) epoch 5774) v5 ==== 0+0+0 (0 0 0) 0x227cbc00 con 0x42c86e0 -2548> 2014-11-08 18:15:34.197518 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 2014-11-08 18:15:34.197486, event: header_read, op: pg_notify(4.21s2(50) epoch 5774) -2547> 2014-11-08 18:15:34.197523 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 0.000000, event: throttled, op: pg_notify(4.21s2(50) epoch 5774) -2546> 2014-11-08 18:15:34.197525 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 0.000000, event: all_read, op: pg_notify(4.21s2(50) epoch 5774) -2545> 2014-11-08 18:15:34.197527 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 2014-11-08 18:15:34.197516, event: dispatched, op: pg_notify(4.21s2(50) epoch 5774) -2544> 2014-11-08 18:15:34.197530 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 2014-11-08 18:15:34.197530, event: waiting_for_osdmap, op: pg_notify(4.21s2(50) epoch 5774) -2543> 2014-11-08 18:15:34.197534 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 2014-11-08 18:15:34.197534, event: started, op: pg_notify(4.21s2(50) epoch 5774) -2542> 2014-11-08 18:15:34.197549 7fc5b9b29700 5 -- op tracker -- seq: 47241, time: 2014-11-08 18:15:34.197549, event: done, op: pg_notify(4.21s2(50) epoch 5774) -2541> 2014-11-08 18:15:34.197576 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.8fs2(21) epoch 5774) v5 ==== 0+0+0 (0 0 0) 0x227ca1c0 con 0x42c86e0 -2540> 2014-11-08 18:15:34.197583 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 2014-11-08 18:15:34.197503, event: header_read, op: pg_notify(4.8fs2(21) epoch 5774) -2539> 2014-11-08 18:15:34.197585 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 0.000000, event: throttled, op: pg_notify(4.8fs2(21) epoch 5774) -2538> 2014-11-08 18:15:34.197587 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 0.000000, event: all_read, op: pg_notify(4.8fs2(21) epoch 5774) -2537> 2014-11-08 18:15:34.197589 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 2014-11-08 18:15:34.197582, event: dispatched, op: pg_notify(4.8fs2(21) epoch 5774) -2536> 2014-11-08 18:15:34.197591 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 2014-11-08 18:15:34.197591, event: waiting_for_osdmap, op: pg_notify(4.8fs2(21) epoch 5774) -2535> 2014-11-08 18:15:34.197595 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 2014-11-08 18:15:34.197595, event: started, op: pg_notify(4.8fs2(21) epoch 5774) -2534> 2014-11-08 18:15:34.197605 7fc5b9b29700 5 -- op tracker -- seq: 47242, time: 2014-11-08 18:15:34.197605, event: done, op: pg_notify(4.8fs2(21) epoch 5774) -2533> 2014-11-08 18:15:34.197632 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.91s1(30) epoch 5774) v5 -- ?+0 0x227c92c0 con 0x1c7911e0 -2532> 2014-11-08 18:15:34.197758 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- MBackfillReserve GRANT pgid: 4.88s0, query_epoch: 5774 v3 -- ?+0 0x227ca940 con 0x1c5c2680 -2531> 2014-11-08 18:15:34.197774 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.88s1( v 5734'8208 (3458'5184,5734'8208] lb 5661fd88/1000002b663.0000000d/head//4 local-les=0 n=6651 ec=195 les/c 5761/5721 5759/5759/5641) [1,9,5]/[1,2147483647,5] r=-1 lpr=5759 pi=4863-5758/39 luod=0'0 crt=5734'8199 active+remapped] exit Started/ReplicaActive/RepWaitBackfillReserved 0.001643 1 0.000046 -2530> 2014-11-08 18:15:34.197792 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.88s1( v 5734'8208 (3458'5184,5734'8208] lb 5661fd88/1000002b663.0000000d/head//4 local-les=0 n=6651 ec=195 les/c 5761/5721 5759/5759/5641) [1,9,5]/[1,2147483647,5] r=-1 lpr=5759 pi=4863-5758/39 luod=0'0 crt=5734'8199 active+remapped] enter Started/ReplicaActive/RepRecovering -2529> 2014-11-08 18:15:34.197812 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MBackfillReserve GRANT pgid: 4.89s0, query_epoch: 5774 v3 -- ?+0 0x227cc560 con 0x1c5c0580 -2528> 2014-11-08 18:15:34.197857 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.89s2( v 5734'8486 (3458'5440,5734'8486] lb 40520589/100000199b6.00000035/head//4 local-les=0 n=2397 ec=195 les/c 5760/5721 5759/5759/5639) [2,5,9]/[2,5,2147483647] r=-1 lpr=5759 pi=5432-5758/33 luod=0'0 crt=5734'8486 active+remapped] exit Started/ReplicaActive/RepWaitBackfillReserved 0.001622 1 0.000031 -2527> 2014-11-08 18:15:34.197874 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.89s2( v 5734'8486 (3458'5440,5734'8486] lb 40520589/100000199b6.00000035/head//4 local-les=0 n=2397 ec=195 les/c 5760/5721 5759/5759/5639) [2,5,9]/[2,5,2147483647] r=-1 lpr=5759 pi=5432-5758/33 luod=0'0 crt=5734'8486 active+remapped] enter Started/ReplicaActive/RepRecovering -2526> 2014-11-08 18:15:34.198253 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8648 ==== MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 v3 ==== 30+0+0 (3924300089 0 0) 0x227c8f00 con 0x1c5c0580 -2525> 2014-11-08 18:15:34.198264 7fc5b9b29700 5 -- op tracker -- seq: 47243, time: 2014-11-08 18:15:34.198211, event: header_read, op: MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 -2524> 2014-11-08 18:15:34.198269 7fc5b9b29700 5 -- op tracker -- seq: 47243, time: 2014-11-08 18:15:34.198212, event: throttled, op: MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 -2523> 2014-11-08 18:15:34.198273 7fc5b9b29700 5 -- op tracker -- seq: 47243, time: 2014-11-08 18:15:34.198244, event: all_read, op: MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 -2522> 2014-11-08 18:15:34.198277 7fc5b9b29700 5 -- op tracker -- seq: 47243, time: 2014-11-08 18:15:34.198263, event: dispatched, op: MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 -2521> 2014-11-08 18:15:34.198281 7fc5b9b29700 5 -- op tracker -- seq: 47243, time: 2014-11-08 18:15:34.198281, event: waiting_for_osdmap, op: MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 -2520> 2014-11-08 18:15:34.198293 7fc5b9b29700 5 -- op tracker -- seq: 47243, time: 2014-11-08 18:15:34.198293, event: done, op: MBackfillReserve GRANT pgid: 4.4bs1, query_epoch: 5774 -2519> 2014-11-08 18:15:34.198304 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4bs1( v 5734'16442 (3473'13377,5734'16442] local-les=5763 n=15306 ec=195 les/c 5763/5636 5759/5759/5728) [2,9,5]/[2147483647,9,5] r=1 lpr=5759 pi=5635-5758/18 bft=2(0) crt=5734'16437 lcod 5734'16441 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitRemoteBackfillReserved 0.001427 1 0.000076 -2518> 2014-11-08 18:15:34.198318 7fc5b1b19700 5 osd.9 pg_epoch: 5774 pg[4.4bs1( v 5734'16442 (3473'13377,5734'16442] local-les=5763 n=15306 ec=195 les/c 5763/5636 5759/5759/5728) [2,9,5]/[2147483647,9,5] r=1 lpr=5759 pi=5635-5758/18 bft=2(0) crt=5734'16437 lcod 5734'16441 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] enter Started/Primary/Active/Backfilling -2517> 2014-11-08 18:15:34.199635 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12089 ==== pg_notify(4.31s2(22),4.36s0(40) epoch 5774) v5 ==== 5156+0+0 (1428429472 0 0) 0x227ca940 con 0x1c5c2680 -2516> 2014-11-08 18:15:34.199647 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199491, event: header_read, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2515> 2014-11-08 18:15:34.199650 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199491, event: throttled, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2514> 2014-11-08 18:15:34.199653 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199558, event: all_read, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2513> 2014-11-08 18:15:34.199656 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199645, event: dispatched, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2512> 2014-11-08 18:15:34.199659 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199659, event: waiting_for_osdmap, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2511> 2014-11-08 18:15:34.199663 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199663, event: started, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2510> 2014-11-08 18:15:34.199683 7fc5b9b29700 5 -- op tracker -- seq: 47244, time: 2014-11-08 18:15:34.199683, event: done, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5774) -2509> 2014-11-08 18:15:34.199711 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12090 ==== pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) v5 ==== 5565+0+0 (4190149005 0 0) 0x227cb0c0 con 0x1c5c2680 -2508> 2014-11-08 18:15:34.199719 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199567, event: header_read, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2507> 2014-11-08 18:15:34.199722 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199568, event: throttled, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2506> 2014-11-08 18:15:34.199725 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199628, event: all_read, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2505> 2014-11-08 18:15:34.199728 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199718, event: dispatched, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2504> 2014-11-08 18:15:34.199731 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199731, event: waiting_for_osdmap, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2503> 2014-11-08 18:15:34.199735 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199735, event: started, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2502> 2014-11-08 18:15:34.199757 7fc5b9b29700 5 -- op tracker -- seq: 47245, time: 2014-11-08 18:15:34.199757, event: done, op: pg_notify(4.39s1(7),4.40s1(30),4.4es2(19) epoch 5774) -2501> 2014-11-08 18:15:34.262854 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDECSubOpRead(4.83s1 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=5b9f1d83/1000002b56e.0000010b/head//4,86ff1d83/1000002b64a.00000041/head//4,42a02d83/1000002b040.00000026/head//4)) v1 -- ?+0 0x2287f980 con 0x1c5c0580 -2500> 2014-11-08 18:15:34.262892 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2287e300 con 0x42c86e0 -2499> 2014-11-08 18:15:34.262906 7fc5abb0d700 5 osd.9 pg_epoch: 5773 pg[4.83s0( v 5734'8135 (3458'5096,5734'8135] local-les=5760 n=7577 ec=195 les/c 5760/5758 5759/5759/5728) [9,2,5]/[9,2,2147483647] r=0 lpr=5759 pi=5757-5758/1 rops=3 bft=5(2) crt=5389'8092 lcod 5734'8134 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is 89e02d83/1000002a94e.00000033/head//4 -2498> 2014-11-08 18:15:34.262918 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2287e300 con 0x42c86e0 -2497> 2014-11-08 18:15:34.262931 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_backfill(progress 4.83s2 e 5773/5773 lb 126f1d83/1000002b7d9.00000099/head//4) v3 -- ?+0 0x1d3c9200 con 0x1e50aec0 -2496> 2014-11-08 18:15:34.262934 7fc5b9328700 5 -- op tracker -- seq: 47246, time: 2014-11-08 18:15:34.262900, event: header_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) -2495> 2014-11-08 18:15:34.262944 7fc5b9328700 5 -- op tracker -- seq: 47246, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) -2494> 2014-11-08 18:15:34.262948 7fc5b9328700 5 -- op tracker -- seq: 47246, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) -2493> 2014-11-08 18:15:34.262951 7fc5b9328700 5 -- op tracker -- seq: 47246, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) -2492> 2014-11-08 18:15:34.263263 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1651.164415 52 0.000213 -2491> 2014-11-08 18:15:34.263278 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active] exit Started/Primary/Active 1665.293166 0 0.000000 -2490> 2014-11-08 18:15:34.263290 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active] exit Started/Primary 1666.370584 0 0.000000 -2489> 2014-11-08 18:15:34.263298 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active] exit Started 1666.370632 0 0.000000 -2488> 2014-11-08 18:15:34.263306 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active] enter Reset -2487> 2014-11-08 18:15:34.263391 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] exit Reset 0.000086 1 0.000102 -2486> 2014-11-08 18:15:34.263402 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] enter Started -2485> 2014-11-08 18:15:34.263408 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] enter Start -2484> 2014-11-08 18:15:34.263414 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2483> 2014-11-08 18:15:34.263423 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] exit Start 0.000015 0 0.000000 -2482> 2014-11-08 18:15:34.263430 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] enter Started/Primary -2481> 2014-11-08 18:15:34.263437 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] enter Started/Primary/Peering -2480> 2014-11-08 18:15:34.263444 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2479> 2014-11-08 18:15:34.263536 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5751/5752/5752) [4,2,9]/[2147483647,2,9] r=2 lpr=5752 pi=2936-5751/76 luod=0'0 crt=5734'8200 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1661.253999 57 0.000330 -2478> 2014-11-08 18:15:34.263548 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5751/5752/5752) [4,2,9]/[2147483647,2,9] r=2 lpr=5752 pi=2936-5751/76 luod=0'0 crt=5734'8200 active+remapped] exit Started/ReplicaActive 1661.254020 0 0.000000 -2477> 2014-11-08 18:15:34.263557 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5751/5752/5752) [4,2,9]/[2147483647,2,9] r=2 lpr=5752 pi=2936-5751/76 luod=0'0 crt=5734'8200 active+remapped] exit Started 1665.962211 0 0.000000 -2476> 2014-11-08 18:15:34.263565 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5751/5752/5752) [4,2,9]/[2147483647,2,9] r=2 lpr=5752 pi=2936-5751/76 luod=0'0 crt=5734'8200 active+remapped] enter Reset -2475> 2014-11-08 18:15:34.263704 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5774/5774/5752) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/77 crt=5734'8200 inactive NOTIFY] exit Reset 0.000139 1 0.000054 -2474> 2014-11-08 18:15:34.263713 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5774/5774/5752) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/77 crt=5734'8200 inactive NOTIFY] enter Started -2473> 2014-11-08 18:15:34.263719 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5774/5774/5752) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/77 crt=5734'8200 inactive NOTIFY] enter Start -2472> 2014-11-08 18:15:34.263724 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5774/5774/5752) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/77 crt=5734'8200 inactive NOTIFY] state<Start>: transitioning to Stray -2471> 2014-11-08 18:15:34.263731 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5774/5774/5752) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/77 crt=5734'8200 inactive NOTIFY] exit Start 0.000012 0 0.000000 -2470> 2014-11-08 18:15:34.263738 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.86s2( v 5734'8206 (3458'5166,5734'8206] local-les=5753 n=7722 ec=195 les/c 5753/5750 5774/5774/5752) [2147483647,2,9] r=2 lpr=5774 pi=2936-5773/77 crt=5734'8200 inactive NOTIFY] enter Started/Stray -2469> 2014-11-08 18:15:34.263838 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/47 luod=0'0 crt=5734'16250 active] exit Started/ReplicaActive/RepNotRecovering 1665.933185 65 0.000371 -2468> 2014-11-08 18:15:34.263848 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/47 luod=0'0 crt=5734'16250 active] exit Started/ReplicaActive 1665.933207 0 0.000000 -2467> 2014-11-08 18:15:34.263856 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/47 luod=0'0 crt=5734'16250 active] exit Started 1666.369876 0 0.000000 -2466> 2014-11-08 18:15:34.263863 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5751/5751/5639) [2,4,9] r=2 lpr=5751 pi=2936-5750/47 luod=0'0 crt=5734'16250 active] enter Reset -2465> 2014-11-08 18:15:34.263933 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/48 crt=5734'16250 inactive NOTIFY] exit Reset 0.000070 1 0.000047 -2464> 2014-11-08 18:15:34.263942 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/48 crt=5734'16250 inactive NOTIFY] enter Started -2463> 2014-11-08 18:15:34.263947 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/48 crt=5734'16250 inactive NOTIFY] enter Start -2462> 2014-11-08 18:15:34.263953 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/48 crt=5734'16250 inactive NOTIFY] state<Start>: transitioning to Stray -2461> 2014-11-08 18:15:34.263960 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/48 crt=5734'16250 inactive NOTIFY] exit Start 0.000013 0 0.000000 -2460> 2014-11-08 18:15:34.263967 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7cs2( v 5734'16260 (3473'13259,5734'16260] local-les=5752 n=15295 ec=195 les/c 5752/5755 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2936-5773/48 crt=5734'16250 inactive NOTIFY] enter Started/Stray -2459> 2014-11-08 18:15:34.264043 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active+clean] exit Started/Primary/Active/Clean 1651.213826 52 0.000205 -2458> 2014-11-08 18:15:34.264053 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active] exit Started/Primary/Active 1665.170677 0 0.000000 -2457> 2014-11-08 18:15:34.264061 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active] exit Started/Primary 1666.369826 0 0.000000 -2456> 2014-11-08 18:15:34.264068 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active] exit Started 1666.369879 0 0.000000 -2455> 2014-11-08 18:15:34.264075 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5751/5751/5729) [9,1,4] r=0 lpr=5751 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active] enter Reset -2454> 2014-11-08 18:15:34.264103 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] exit Reset 0.000028 1 0.000060 -2453> 2014-11-08 18:15:34.264112 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] enter Started -2452> 2014-11-08 18:15:34.264118 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] enter Start -2451> 2014-11-08 18:15:34.264124 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2450> 2014-11-08 18:15:34.264132 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] exit Start 0.000013 0 0.000000 -2449> 2014-11-08 18:15:34.264139 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] enter Started/Primary -2448> 2014-11-08 18:15:34.264145 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] enter Started/Primary/Peering -2447> 2014-11-08 18:15:34.264152 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2446> 2014-11-08 18:15:34.264223 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16391 active] exit Started/ReplicaActive/RepNotRecovering 1661.254371 58 0.000292 -2445> 2014-11-08 18:15:34.264233 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16391 active] exit Started/ReplicaActive 1661.254389 0 0.000000 -2444> 2014-11-08 18:15:34.264241 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16391 active] exit Started 1666.368348 0 0.000000 -2443> 2014-11-08 18:15:34.264248 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5751/5751/5641) [1,4,9] r=2 lpr=5751 pi=4863-5750/33 luod=0'0 crt=5734'16391 active] enter Reset -2442> 2014-11-08 18:15:34.264301 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] exit Reset 0.000053 1 0.000047 -2441> 2014-11-08 18:15:34.264309 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] enter Started -2440> 2014-11-08 18:15:34.264327 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] enter Start -2439> 2014-11-08 18:15:34.264333 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] state<Start>: transitioning to Stray -2438> 2014-11-08 18:15:34.264340 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] exit Start 0.000013 0 0.000000 -2437> 2014-11-08 18:15:34.264347 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] enter Started/Stray -2436> 2014-11-08 18:15:34.264411 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5751/5751/5639) [2,4,9]/[2,2147483647,9] r=2 lpr=5751 pi=2960-5750/73 luod=0'0 crt=5734'16524 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1665.933360 64 0.000371 -2435> 2014-11-08 18:15:34.264421 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5751/5751/5639) [2,4,9]/[2,2147483647,9] r=2 lpr=5751 pi=2960-5750/73 luod=0'0 crt=5734'16524 active+remapped] exit Started/ReplicaActive 1665.933382 0 0.000000 -2434> 2014-11-08 18:15:34.264429 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5751/5751/5639) [2,4,9]/[2,2147483647,9] r=2 lpr=5751 pi=2960-5750/73 luod=0'0 crt=5734'16524 active+remapped] exit Started 1666.368291 0 0.000000 -2433> 2014-11-08 18:15:34.264437 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5751/5751/5639) [2,4,9]/[2,2147483647,9] r=2 lpr=5751 pi=2960-5750/73 luod=0'0 crt=5734'16524 active+remapped] enter Reset -2432> 2014-11-08 18:15:34.264577 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2960-5773/74 crt=5734'16524 inactive NOTIFY] exit Reset 0.000140 1 0.000049 -2431> 2014-11-08 18:15:34.264585 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2960-5773/74 crt=5734'16524 inactive NOTIFY] enter Started -2430> 2014-11-08 18:15:34.264591 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2960-5773/74 crt=5734'16524 inactive NOTIFY] enter Start -2429> 2014-11-08 18:15:34.264597 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2960-5773/74 crt=5734'16524 inactive NOTIFY] state<Start>: transitioning to Stray -2428> 2014-11-08 18:15:34.264604 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2960-5773/74 crt=5734'16524 inactive NOTIFY] exit Start 0.000013 0 0.000000 -2427> 2014-11-08 18:15:34.264610 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.61s2( v 5734'16527 (3473'13441,5734'16527] local-les=5752 n=15454 ec=195 les/c 5752/5750 5774/5774/5639) [2,2147483647,9] r=2 lpr=5774 pi=2960-5773/74 crt=5734'16524 inactive NOTIFY] enter Started/Stray -2426> 2014-11-08 18:15:34.264741 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=-1 lpr=5751 pi=4872-5750/28 luod=0'0 crt=5734'16521 active+remapped] exit Started/ReplicaActive/RepRecovering 1026.059498 49 0.000153 -2425> 2014-11-08 18:15:34.264754 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=-1 lpr=5751 pi=4872-5750/28 luod=0'0 crt=5734'16521 active+remapped] exit Started/ReplicaActive 1661.251179 0 0.000000 -2424> 2014-11-08 18:15:34.264763 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=-1 lpr=5751 pi=4872-5750/28 luod=0'0 crt=5734'16521 active+remapped] exit Started 1666.368219 0 0.000000 -2423> 2014-11-08 18:15:34.264771 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=-1 lpr=5751 pi=4872-5750/28 luod=0'0 crt=5734'16521 active+remapped] enter Reset -2422> 2014-11-08 18:15:34.264810 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] exit Reset 0.000039 1 0.000052 -2421> 2014-11-08 18:15:34.264820 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] enter Started -2420> 2014-11-08 18:15:34.264827 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] enter Start -2419> 2014-11-08 18:15:34.264833 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] state<Start>: transitioning to Stray -2418> 2014-11-08 18:15:34.264840 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] exit Start 0.000013 0 0.000000 -2417> 2014-11-08 18:15:34.264848 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] enter Started/Stray -2416> 2014-11-08 18:15:34.264903 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=2 lpr=5751 pi=2951-5750/77 luod=0'0 crt=5734'16515 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1661.253618 57 0.002789 -2415> 2014-11-08 18:15:34.264913 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=2 lpr=5751 pi=2951-5750/77 luod=0'0 crt=5734'16515 active+remapped] exit Started/ReplicaActive 1661.253636 0 0.000000 -2414> 2014-11-08 18:15:34.264921 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=2 lpr=5751 pi=2951-5750/77 luod=0'0 crt=5734'16515 active+remapped] exit Started 1666.367988 0 0.000000 -2413> 2014-11-08 18:15:34.264929 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5751/5751/5641) [1,9,4]/[2147483647,1,9] r=2 lpr=5751 pi=2951-5750/77 luod=0'0 crt=5734'16515 active+remapped] enter Reset -2412> 2014-11-08 18:15:34.264999 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] exit Reset 0.000070 1 0.000046 -2411> 2014-11-08 18:15:34.265009 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] enter Started -2410> 2014-11-08 18:15:34.265015 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] enter Start -2409> 2014-11-08 18:15:34.265021 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] state<Start>: transitioning to Stray -2408> 2014-11-08 18:15:34.265028 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] exit Start 0.000013 0 0.000000 -2407> 2014-11-08 18:15:34.265035 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] enter Started/Stray -2406> 2014-11-08 18:15:34.265113 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'16313 active] exit Started/ReplicaActive/RepNotRecovering 1665.958449 65 0.000280 -2405> 2014-11-08 18:15:34.265123 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'16313 active] exit Started/ReplicaActive 1665.958467 0 0.000000 -2404> 2014-11-08 18:15:34.265131 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'16313 active] exit Started 1666.367909 0 0.000000 -2403> 2014-11-08 18:15:34.265138 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5751/5751/5751) [4,9,2] r=1 lpr=5751 pi=5749-5750/1 luod=0'0 crt=5734'16313 active] enter Reset -2402> 2014-11-08 18:15:34.265162 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] exit Reset 0.000024 1 0.000048 -2401> 2014-11-08 18:15:34.265171 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] enter Started -2400> 2014-11-08 18:15:34.265176 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] enter Start -2399> 2014-11-08 18:15:34.265182 7fc5b1318700 1 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] state<Start>: transitioning to Primary -2398> 2014-11-08 18:15:34.265189 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] exit Start 0.000013 0 0.000000 -2397> 2014-11-08 18:15:34.265203 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] enter Started/Primary -2396> 2014-11-08 18:15:34.265209 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] enter Started/Primary/Peering -2395> 2014-11-08 18:15:34.265215 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] enter Started/Primary/Peering/GetInfo -2394> 2014-11-08 18:15:34.265254 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.8ds1(11),4.6ds2(34),4.67s1(29),4.67s2(78) epoch 5774) v5 -- ?+0 0x2127a580 con 0x1c5c2680 -2393> 2014-11-08 18:15:34.265268 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.86s2(77),4.7cs2(48),4.61s2(74) epoch 5774) v5 -- ?+0 0x2127a3a0 con 0x1c5c0580 -2392> 2014-11-08 18:15:34.265277 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_query(4.75s1,4.81s1 epoch 5774) v3 -- ?+0 0x2127a1c0 con 0x1c5c2680 -2391> 2014-11-08 18:15:34.265283 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_query(4.65s2 epoch 5774) v3 -- ?+0 0x21279fe0 con 0x1c5c0580 -2390> 2014-11-08 18:15:34.265309 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.001864 2 0.000074 -2389> 2014-11-08 18:15:34.265319 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2388> 2014-11-08 18:15:34.265475 7fc5b0316700 5 -- op tracker -- seq: 47246, time: 2014-11-08 18:15:34.265475, event: reached_pg, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) -2387> 2014-11-08 18:15:34.265339 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000021 0 0.000000 -2386> 2014-11-08 18:15:34.266313 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2385> 2014-11-08 18:15:34.266324 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000011 0 0.000000 -2384> 2014-11-08 18:15:34.266333 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2383> 2014-11-08 18:15:34.266368 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.002216 2 0.000061 -2382> 2014-11-08 18:15:34.266378 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2381> 2014-11-08 18:15:34.266395 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000017 0 0.000000 -2380> 2014-11-08 18:15:34.266438 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2379> 2014-11-08 18:15:34.266446 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000008 0 0.000000 -2378> 2014-11-08 18:15:34.266455 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2377> 2014-11-08 18:15:34.266662 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7bs0( v 5734'16429 (3473'13366,5734'16429] local-les=5761 n=15368 ec=195 les/c 5761/5636 5759/5759/5728) [9,1,5]/[9,2147483647,5] r=0 lpr=5759 pi=5635-5758/18 bft=1(1) crt=5734'16415 lcod 5734'16428 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] exit Started/Primary/Active/WaitRemoteBackfillReserved 720.841909 22 0.000107 -2376> 2014-11-08 18:15:34.266681 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.7bs0( v 5734'16429 (3473'13366,5734'16429] local-les=5761 n=15368 ec=195 les/c 5761/5636 5759/5759/5728) [9,1,5]/[9,2147483647,5] r=0 lpr=5759 pi=5635-5758/18 bft=1(1) crt=5734'16415 lcod 5734'16428 mlcod 0'0 active+undersized+degraded+remapped+wait_backfill] enter Started/Primary/Active/Backfilling -2375> 2014-11-08 18:15:34.266749 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] exit Started/Primary/Peering/GetInfo 0.001534 2 0.000067 -2374> 2014-11-08 18:15:34.266758 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] enter Started/Primary/Peering/GetLog -2373> 2014-11-08 18:15:34.266774 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] exit Started/Primary/Peering/GetLog 0.000016 0 0.000000 -2372> 2014-11-08 18:15:34.266783 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] enter Started/Primary/Peering/GetMissing -2371> 2014-11-08 18:15:34.266791 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.000008 0 0.000000 -2370> 2014-11-08 18:15:34.266799 7fc5b1318700 5 osd.9 pg_epoch: 5774 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] enter Started/Primary/Peering/WaitUpThru -2369> 2014-11-08 18:15:34.266814 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.8ds1(11),4.6ds2(34),4.67s1(29),4.67s2(78) epoch 5774) v5 -- ?+0 0x21279a40 con 0x1c5c2680 -2368> 2014-11-08 18:15:34.266825 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.61s2(74),4.7cs2(48),4.86s2(77) epoch 5774) v5 -- ?+0 0x21279860 con 0x1c5c0580 -2367> 2014-11-08 18:15:34.267748 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2296 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x22acf080 con 0x1e50aec0 -2366> 2014-11-08 18:15:34.267768 7fc5b9b29700 3 osd.9 5774 handle_osd_map epochs [5774,5774], i have 5774, src has [4761,5774] -2365> 2014-11-08 18:15:34.269287 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8649 ==== pg_notify(4.65s2(68) epoch 5774) v5 ==== 4767+0+0 (712349338 0 0) 0x21279860 con 0x1c5c0580 -2364> 2014-11-08 18:15:34.269299 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269192, event: header_read, op: pg_notify(4.65s2(68) epoch 5774) -2363> 2014-11-08 18:15:34.269305 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269193, event: throttled, op: pg_notify(4.65s2(68) epoch 5774) -2362> 2014-11-08 18:15:34.269308 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269275, event: all_read, op: pg_notify(4.65s2(68) epoch 5774) -2361> 2014-11-08 18:15:34.269310 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269297, event: dispatched, op: pg_notify(4.65s2(68) epoch 5774) -2360> 2014-11-08 18:15:34.269313 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269313, event: waiting_for_osdmap, op: pg_notify(4.65s2(68) epoch 5774) -2359> 2014-11-08 18:15:34.269320 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269320, event: started, op: pg_notify(4.65s2(68) epoch 5774) -2358> 2014-11-08 18:15:34.269338 7fc5b9b29700 5 -- op tracker -- seq: 47247, time: 2014-11-08 18:15:34.269338, event: done, op: pg_notify(4.65s2(68) epoch 5774) -2357> 2014-11-08 18:15:34.270780 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12091 ==== pg_notify(4.75s1(7),4.81s1(7) epoch 5774) v5 ==== 2324+0+0 (3505871486 0 0) 0x21279a40 con 0x1c5c2680 -2356> 2014-11-08 18:15:34.270792 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270718, event: header_read, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2355> 2014-11-08 18:15:34.270796 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270719, event: throttled, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2354> 2014-11-08 18:15:34.270799 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270769, event: all_read, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2353> 2014-11-08 18:15:34.270801 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270790, event: dispatched, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2352> 2014-11-08 18:15:34.270805 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270805, event: waiting_for_osdmap, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2351> 2014-11-08 18:15:34.270809 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270809, event: started, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2350> 2014-11-08 18:15:34.270827 7fc5b9b29700 5 -- op tracker -- seq: 47248, time: 2014-11-08 18:15:34.270827, event: done, op: pg_notify(4.75s1(7),4.81s1(7) epoch 5774) -2349> 2014-11-08 18:15:34.279728 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 334 ==== pg_query(4.5ds0 epoch 5774) v3 ==== 144+0+0 (2554334640 0 0) 0x227c92c0 con 0x1c7911e0 -2348> 2014-11-08 18:15:34.279747 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279629, event: header_read, op: pg_query(4.5ds0 epoch 5774) -2347> 2014-11-08 18:15:34.279753 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279632, event: throttled, op: pg_query(4.5ds0 epoch 5774) -2346> 2014-11-08 18:15:34.279756 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279705, event: all_read, op: pg_query(4.5ds0 epoch 5774) -2345> 2014-11-08 18:15:34.279758 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279745, event: dispatched, op: pg_query(4.5ds0 epoch 5774) -2344> 2014-11-08 18:15:34.279761 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279761, event: waiting_for_osdmap, op: pg_query(4.5ds0 epoch 5774) -2343> 2014-11-08 18:15:34.279766 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279766, event: started, op: pg_query(4.5ds0 epoch 5774) -2342> 2014-11-08 18:15:34.279781 7fc5b9b29700 5 -- op tracker -- seq: 47249, time: 2014-11-08 18:15:34.279781, event: done, op: pg_query(4.5ds0 epoch 5774) -2341> 2014-11-08 18:15:34.279977 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.5ds0(91) epoch 5774) v5 -- ?+0 0x21279680 con 0x1c7911e0 -2340> 2014-11-08 18:15:34.382418 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDECSubOpRead(4.43s1 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=5981f343/1000002ab09.00000020/head//4,d7c1f343/1000002aa96.00000035/head//4)) v1 -- ?+0 0x2287e540 con 0x1e50aec0 -2339> 2014-11-08 18:15:34.382457 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) v1 -- ?+0 0x2287de80 con 0x42c86e0 -2338> 2014-11-08 18:15:34.382469 7fc5abb0d700 5 osd.9 pg_epoch: 5774 pg[4.43s0( v 5734'16291 (3473'13241,5734'16291] local-les=5760 n=15175 ec=195 les/c 5760/5636 5759/5759/5728) [9,5,2]/[9,5,2147483647] r=0 lpr=5759 pi=5635-5758/19 rops=2 bft=2(2) crt=5734'16286 lcod 5734'16290 mlcod 0'0 active+undersized+degraded+remapped+backfilling] backfill_pos is f5f1f343/1000002b0f9.000000f0/head//4 -2337> 2014-11-08 18:15:34.382496 7fc5abb0d700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_backfill(progress 4.43s2 e 5774/5774 lb b931f343/1000001a121.0000017c/head//4) v3 -- ?+0 0x23ceec00 con 0x1c5c0580 -2336> 2014-11-08 18:15:34.382535 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) v1 ==== 0+0+0 (0 0 0) 0x2287de80 con 0x42c86e0 -2335> 2014-11-08 18:15:34.382548 7fc5b9328700 5 -- op tracker -- seq: 47250, time: 2014-11-08 18:15:34.382464, event: header_read, op: MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) -2334> 2014-11-08 18:15:34.382559 7fc5b9328700 5 -- op tracker -- seq: 47250, time: 0.000000, event: throttled, op: MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) -2333> 2014-11-08 18:15:34.382562 7fc5b9328700 5 -- op tracker -- seq: 47250, time: 0.000000, event: all_read, op: MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) -2332> 2014-11-08 18:15:34.382565 7fc5b9328700 5 -- op tracker -- seq: 47250, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) -2331> 2014-11-08 18:15:34.382606 7fc5afb15700 5 -- op tracker -- seq: 47250, time: 2014-11-08 18:15:34.382606, event: reached_pg, op: MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) -2330> 2014-11-08 18:15:34.437567 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1510 ==== osd_map(5774..5775 src has 4761..5775) v3 ==== 477+0+0 (121689205 0 0) 0x22ac8b40 con 0x44ed6a0 -2329> 2014-11-08 18:15:34.437612 7fc5bb32c700 3 osd.9 5774 handle_osd_map epochs [5774,5775], i have 5774, src has [4761,5775] -2328> 2014-11-08 18:15:34.464347 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 335 ==== pg_notify(4.36s2(40) epoch 5774) v5 ==== 3135+0+0 (150075666 0 0) 0x21279680 con 0x1c7911e0 -2327> 2014-11-08 18:15:34.530874 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) v1 -- ?+0 0x1d5fc100 con 0x42c86e0 -2326> 2014-11-08 18:15:34.530922 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x1d5fc100 con 0x42c86e0 -2325> 2014-11-08 18:15:34.530935 7fc5b9328700 5 -- op tracker -- seq: 47251, time: 2014-11-08 18:15:34.530911, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) -2324> 2014-11-08 18:15:34.530943 7fc5b9328700 5 -- op tracker -- seq: 47251, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) -2323> 2014-11-08 18:15:34.530945 7fc5b9328700 5 -- op tracker -- seq: 47251, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) -2322> 2014-11-08 18:15:34.530947 7fc5b9328700 5 -- op tracker -- seq: 47251, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) -2321> 2014-11-08 18:15:34.530993 7fc5ad310700 5 -- op tracker -- seq: 47251, time: 2014-11-08 18:15:34.530993, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) -2320> 2014-11-08 18:15:34.531035 7fc5ad310700 5 -- op tracker -- seq: 47251, time: 2014-11-08 18:15:34.531035, event: done, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=0)) -2319> 2014-11-08 18:15:34.531055 7fc5afb15700 5 -- op tracker -- seq: 47250, time: 2014-11-08 18:15:34.531055, event: done, op: MOSDECSubOpRead(4.43s0 5774 ECSubRead(tid=4428, to_read={5981f343/1000002ab09.00000020/head//4=0,4194304,d7c1f343/1000002aa96.00000035/head//4=0,4194304}, attrs_to_read=)) -2318> 2014-11-08 18:15:34.781410 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8651 ==== MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) v1 ==== 6292958+0+0 (3752791276 0 0) 0x1c89b480 con 0x1c5c0580 -2317> 2014-11-08 18:15:34.781452 7fc5a152e700 5 -- op tracker -- seq: 47252, time: 2014-11-08 18:15:34.602944, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) -2316> 2014-11-08 18:15:34.781466 7fc5a152e700 5 -- op tracker -- seq: 47252, time: 2014-11-08 18:15:34.602947, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) -2315> 2014-11-08 18:15:34.781475 7fc5a152e700 5 -- op tracker -- seq: 47252, time: 2014-11-08 18:15:34.781375, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) -2314> 2014-11-08 18:15:34.781484 7fc5a152e700 5 -- op tracker -- seq: 47252, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) -2313> 2014-11-08 18:15:34.895865 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8652 ==== MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 6294102+0+0 (1441137194 0 0) 0x1d3a9400 con 0x1c5c0580 -2312> 2014-11-08 18:15:34.895920 7fc5a152e700 5 -- op tracker -- seq: 47253, time: 2014-11-08 18:15:34.833202, event: header_read, op: MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -2311> 2014-11-08 18:15:34.895944 7fc5a152e700 5 -- op tracker -- seq: 47253, time: 2014-11-08 18:15:34.833205, event: throttled, op: MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -2310> 2014-11-08 18:15:34.895963 7fc5a152e700 5 -- op tracker -- seq: 47253, time: 2014-11-08 18:15:34.895833, event: all_read, op: MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -2309> 2014-11-08 18:15:34.895981 7fc5a152e700 5 -- op tracker -- seq: 47253, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -2308> 2014-11-08 18:15:34.896045 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- osd_map(5775..5775 src has 4761..5775) v3 -- ?+0 0x1ce618c0 con 0x1c5c0580 -2307> 2014-11-08 18:15:34.896056 7fc5ad310700 5 -- op tracker -- seq: 47253, time: 2014-11-08 18:15:34.896056, event: reached_pg, op: MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -2306> 2014-11-08 18:15:34.896251 7fc5a152e700 5 -- op tracker -- seq: 47253, time: 2014-11-08 18:15:34.896251, event: done, op: MOSDPGPush(4.16s2 5774 [PushOp(1add9b16/10000028733.00000138/head//4, version: 3033'3960, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(1add9b16/10000028733.00000138/head//4@3033'3960, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(a63e9b16/1000001acec.0000002d/head//4, version: 2829'2623, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(a63e9b16/1000001acec.0000002d/head//4@2829'2623, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5c5e9b16/1000001a265.0000005f/head//4, version: 397'1844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5c5e9b16/1000001a265.0000005f/head//4@397'1844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -2305> 2014-11-08 18:15:35.365626 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) v1 -- ?+0 0x224bf080 con 0x42c86e0 -2304> 2014-11-08 18:15:35.365674 7fc5b9328700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) v1 ==== 0+0+0 (0 0 0) 0x224bf080 con 0x42c86e0 -2303> 2014-11-08 18:15:35.365694 7fc5b9328700 5 -- op tracker -- seq: 47254, time: 2014-11-08 18:15:35.365661, event: header_read, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) -2302> 2014-11-08 18:15:35.365704 7fc5b9328700 5 -- op tracker -- seq: 47254, time: 0.000000, event: throttled, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) -2301> 2014-11-08 18:15:35.365705 7fc5b9328700 5 -- op tracker -- seq: 47254, time: 0.000000, event: all_read, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) -2300> 2014-11-08 18:15:35.365707 7fc5b9328700 5 -- op tracker -- seq: 47254, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) -2299> 2014-11-08 18:15:35.365941 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5774) v1 -- ?+0 0x1f4c5280 con 0x1c5c0580 -2298> 2014-11-08 18:15:35.365955 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5774) v1 -- ?+0 0x1e897080 con 0x1c5c0580 -2297> 2014-11-08 18:15:35.365962 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_trim(4.16s0 to 5734'8390 e5774) v1 -- ?+0 0x1e8932a0 con 0x1c5c0580 -2296> 2014-11-08 18:15:35.365967 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPushReply(4.16s0 5774 [PushReplyOp(1add9b16/10000028733.00000138/head//4),PushReplyOp(a63e9b16/1000001acec.0000002d/head//4),PushReplyOp(5c5e9b16/1000001a265.0000005f/head//4)]) v2 -- ?+0 0x229be000 con 0x1c5c0580 -2295> 2014-11-08 18:15:35.365995 7fc5adb11700 5 -- op tracker -- seq: 47252, time: 2014-11-08 18:15:35.365995, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) -2294> 2014-11-08 18:15:35.366047 7fc5adb11700 5 -- op tracker -- seq: 47252, time: 2014-11-08 18:15:35.366047, event: done, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=3)) -2293> 2014-11-08 18:15:35.366098 7fc5adb11700 5 -- op tracker -- seq: 47254, time: 2014-11-08 18:15:35.366098, event: reached_pg, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) -2292> 2014-11-08 18:15:35.367428 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- osd_map(5775..5775 src has 4761..5775) v3 -- ?+0 0x224aa1c0 con 0x1c5c2680 -2291> 2014-11-08 18:15:35.367448 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.5as2(93),4.57s1(69),4.45s2(34) epoch 5775) v5 -- ?+0 0x212794a0 con 0x1c5c2680 -2290> 2014-11-08 18:15:35.367456 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.5bs1(11),4.49s2(54),4.3fs0(38),4.33s2(44) epoch 5775) v5 -- ?+0 0x212792c0 con 0x1c5c0580 -2289> 2014-11-08 18:15:35.367464 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- osd_map(5775..5775 src has 4761..5775) v3 -- ?+0 0x224a8480 con 0x1c7911e0 -2288> 2014-11-08 18:15:35.367473 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.5ds0(91) epoch 5775) v5 -- ?+0 0x212790e0 con 0x1c7911e0 -2287> 2014-11-08 18:15:35.369004 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.24s1(11),4.12s1(53),4.99s1(11),4.8ds1(11),4.9fs2(42),4.59s2(61) epoch 5775) v5 -- ?+0 0x227c9a40 con 0x1c5c2680 -2286> 2014-11-08 18:15:35.369028 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.2ds2(46),4.14s2(44),4.bcs1(76),4.bds1(19),4.b2s2(71),4.b5s1(22),4.9cs2(22),4.2as2(35),4.94s1(11) epoch 5775) v5 -- ?+0 0x227c8d20 con 0x1c5c0580 -2285> 2014-11-08 18:15:35.369052 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.21s2(50) epoch 5775) v5 -- ?+0 0x227cdbe0 con 0x42c86e0 -2284> 2014-11-08 18:15:35.369064 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.8fs2(21) epoch 5775) v5 -- ?+0 0x234e3de0 con 0x42c86e0 -2283> 2014-11-08 18:15:35.369071 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.91s1(30) epoch 5775) v5 -- ?+0 0x234e2d00 con 0x1c7911e0 -2282> 2014-11-08 18:15:35.369434 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.67s1(29),4.67s2(78),4.ds1(11),4.7s2(24) epoch 5775) v5 -- ?+0 0x234e6360 con 0x1c5c2680 -2281> 2014-11-08 18:15:35.369444 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.61s2(74),4.6s2(24) epoch 5775) v5 -- ?+0 0x234e6cc0 con 0x1c5c0580 -2280> 2014-11-08 18:15:35.375530 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.6ds2(34) epoch 5775) v5 -- ?+0 0x21278f00 con 0x1c5c2680 -2279> 2014-11-08 18:15:35.375567 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.86s2(77),4.7cs2(48) epoch 5775) v5 -- ?+0 0x21279c20 con 0x1c5c0580 -2278> 2014-11-08 18:15:35.377440 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- osd_map(5775..5775 src has 4761..5775) v3 -- ?+0 0x22acc380 con 0x1e50aec0 -2277> 2014-11-08 18:15:35.377459 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_notify(4.a9s0(45) epoch 5775) v5 -- ?+0 0x234e50a0 con 0x1e50aec0 -2276> 2014-11-08 18:15:35.379634 7fc5b0316700 5 -- op tracker -- seq: 47246, time: 2014-11-08 18:15:35.379634, event: done, op: MOSDECSubOpRead(4.83s0 5773 ECSubRead(tid=4427, to_read={5b9f1d83/1000002b56e.0000010b/head//4=0,4194304,86ff1d83/1000002b64a.00000041/head//4=0,4194304,42a02d83/1000002b040.00000026/head//4=0,4194304}, attrs_to_read=)) -2275> 2014-11-08 18:15:35.404240 7fc5adb11700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MOSDPGPush(4.83s2 5774 [PushOp(5b9f1d83/1000002b56e.0000010b/head//4, version: 3473'6331, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5b9f1d83/1000002b56e.0000010b/head//4@3473'6331, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(86ff1d83/1000002b64a.00000041/head//4, version: 3473'6511, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(86ff1d83/1000002b64a.00000041/head//4@3473'6511, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(42a02d83/1000002b040.00000026/head//4, version: 4157'7654, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(42a02d83/1000002b040.00000026/head//4@4157'7654, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3af200 con 0x1e50aec0 -2274> 2014-11-08 18:15:35.414266 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:34.464240, event: header_read, op: pg_notify(4.36s2(40) epoch 5774) -2273> 2014-11-08 18:15:35.414283 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:34.464243, event: throttled, op: pg_notify(4.36s2(40) epoch 5774) -2272> 2014-11-08 18:15:35.414286 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:34.464322, event: all_read, op: pg_notify(4.36s2(40) epoch 5774) -2271> 2014-11-08 18:15:35.414289 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:34.464366, event: dispatched, op: pg_notify(4.36s2(40) epoch 5774) -2270> 2014-11-08 18:15:35.414292 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:35.414292, event: waiting_for_osdmap, op: pg_notify(4.36s2(40) epoch 5774) -2269> 2014-11-08 18:15:35.414301 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:35.414301, event: started, op: pg_notify(4.36s2(40) epoch 5774) -2268> 2014-11-08 18:15:35.414321 7fc5b9b29700 5 -- op tracker -- seq: 47255, time: 2014-11-08 18:15:35.414321, event: done, op: pg_notify(4.36s2(40) epoch 5774) -2267> 2014-11-08 18:15:35.414635 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2297 ==== pg_notify(4.21s0(100) epoch 5774) v5 ==== 6675+0+0 (624284957 0 0) 0x213beae0 con 0x1e50aec0 -2266> 2014-11-08 18:15:35.414653 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:34.492862, event: header_read, op: pg_notify(4.21s0(100) epoch 5774) -2265> 2014-11-08 18:15:35.414656 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:34.492864, event: throttled, op: pg_notify(4.21s0(100) epoch 5774) -2264> 2014-11-08 18:15:35.414659 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:34.493119, event: all_read, op: pg_notify(4.21s0(100) epoch 5774) -2263> 2014-11-08 18:15:35.414661 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:35.414651, event: dispatched, op: pg_notify(4.21s0(100) epoch 5774) -2262> 2014-11-08 18:15:35.414664 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:35.414664, event: waiting_for_osdmap, op: pg_notify(4.21s0(100) epoch 5774) -2261> 2014-11-08 18:15:35.414668 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:35.414668, event: started, op: pg_notify(4.21s0(100) epoch 5774) -2260> 2014-11-08 18:15:35.414680 7fc5b9b29700 5 -- op tracker -- seq: 47256, time: 2014-11-08 18:15:35.414680, event: done, op: pg_notify(4.21s0(100) epoch 5774) -2259> 2014-11-08 18:15:35.414734 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.21s2(50) epoch 5775) v5 ==== 0+0+0 (0 0 0) 0x227cdbe0 con 0x42c86e0 -2258> 2014-11-08 18:15:35.414743 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 2014-11-08 18:15:35.369058, event: header_read, op: pg_notify(4.21s2(50) epoch 5775) -2257> 2014-11-08 18:15:35.414747 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 0.000000, event: throttled, op: pg_notify(4.21s2(50) epoch 5775) -2256> 2014-11-08 18:15:35.414749 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 0.000000, event: all_read, op: pg_notify(4.21s2(50) epoch 5775) -2255> 2014-11-08 18:15:35.414751 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 2014-11-08 18:15:35.414742, event: dispatched, op: pg_notify(4.21s2(50) epoch 5775) -2254> 2014-11-08 18:15:35.414754 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 2014-11-08 18:15:35.414753, event: waiting_for_osdmap, op: pg_notify(4.21s2(50) epoch 5775) -2253> 2014-11-08 18:15:35.414757 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 2014-11-08 18:15:35.414757, event: started, op: pg_notify(4.21s2(50) epoch 5775) -2252> 2014-11-08 18:15:35.414768 7fc5b9b29700 5 -- op tracker -- seq: 47257, time: 2014-11-08 18:15:35.414768, event: done, op: pg_notify(4.21s2(50) epoch 5775) -2251> 2014-11-08 18:15:35.414784 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8650 ==== osd_map(5775..5775 src has 4761..5775) v3 ==== 298+0+0 (4097883411 0 0) 0x2287f980 con 0x1c5c0580 -2250> 2014-11-08 18:15:35.414800 7fc5b9b29700 3 osd.9 5775 handle_osd_map epochs [5775,5775], i have 5775, src has [4761,5775] -2249> 2014-11-08 18:15:35.414806 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12092 ==== osd_map(5775..5775 src has 4761..5775) v3 ==== 298+0+0 (4097883411 0 0) 0x22acc800 con 0x1c5c2680 -2248> 2014-11-08 18:15:35.414813 7fc5b9b29700 3 osd.9 5775 handle_osd_map epochs [5775,5775], i have 5775, src has [4761,5775] -2247> 2014-11-08 18:15:35.414818 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 336 ==== pg_notify(4.15s1(28) epoch 5774) v5 ==== 2427+0+0 (3285636965 0 0) 0x21221860 con 0x1c7911e0 -2246> 2014-11-08 18:15:35.414824 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:34.515934, event: header_read, op: pg_notify(4.15s1(28) epoch 5774) -2245> 2014-11-08 18:15:35.414827 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:34.515937, event: throttled, op: pg_notify(4.15s1(28) epoch 5774) -2244> 2014-11-08 18:15:35.414830 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:34.516025, event: all_read, op: pg_notify(4.15s1(28) epoch 5774) -2243> 2014-11-08 18:15:35.414832 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:35.414824, event: dispatched, op: pg_notify(4.15s1(28) epoch 5774) -2242> 2014-11-08 18:15:35.414835 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:35.414834, event: waiting_for_osdmap, op: pg_notify(4.15s1(28) epoch 5774) -2241> 2014-11-08 18:15:35.414838 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:35.414838, event: started, op: pg_notify(4.15s1(28) epoch 5774) -2240> 2014-11-08 18:15:35.414848 7fc5b9b29700 5 -- op tracker -- seq: 47258, time: 2014-11-08 18:15:35.414848, event: done, op: pg_notify(4.15s1(28) epoch 5774) -2239> 2014-11-08 18:15:35.414860 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2298 ==== pg_notify(4.bfs2(127) epoch 5774) v5 ==== 8248+0+0 (3816182600 0 0) 0x1dbf9c20 con 0x1e50aec0 -2238> 2014-11-08 18:15:35.414867 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.168798, event: header_read, op: pg_notify(4.bfs2(127) epoch 5774) -2237> 2014-11-08 18:15:35.414870 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.168801, event: throttled, op: pg_notify(4.bfs2(127) epoch 5774) -2236> 2014-11-08 18:15:35.414872 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.169007, event: all_read, op: pg_notify(4.bfs2(127) epoch 5774) -2235> 2014-11-08 18:15:35.414875 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.414866, event: dispatched, op: pg_notify(4.bfs2(127) epoch 5774) -2234> 2014-11-08 18:15:35.414877 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.414877, event: waiting_for_osdmap, op: pg_notify(4.bfs2(127) epoch 5774) -2233> 2014-11-08 18:15:35.414880 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.414880, event: started, op: pg_notify(4.bfs2(127) epoch 5774) -2232> 2014-11-08 18:15:35.414890 7fc5b9b29700 5 -- op tracker -- seq: 47259, time: 2014-11-08 18:15:35.414890, event: done, op: pg_notify(4.bfs2(127) epoch 5774) -2231> 2014-11-08 18:15:35.414969 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.8fs2(21) epoch 5775) v5 ==== 0+0+0 (0 0 0) 0x234e3de0 con 0x42c86e0 -2230> 2014-11-08 18:15:35.414976 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 2014-11-08 18:15:35.369069, event: header_read, op: pg_notify(4.8fs2(21) epoch 5775) -2229> 2014-11-08 18:15:35.414978 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 0.000000, event: throttled, op: pg_notify(4.8fs2(21) epoch 5775) -2228> 2014-11-08 18:15:35.414980 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 0.000000, event: all_read, op: pg_notify(4.8fs2(21) epoch 5775) -2227> 2014-11-08 18:15:35.414981 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 2014-11-08 18:15:35.414975, event: dispatched, op: pg_notify(4.8fs2(21) epoch 5775) -2226> 2014-11-08 18:15:35.414984 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 2014-11-08 18:15:35.414984, event: waiting_for_osdmap, op: pg_notify(4.8fs2(21) epoch 5775) -2225> 2014-11-08 18:15:35.414987 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 2014-11-08 18:15:35.414987, event: started, op: pg_notify(4.8fs2(21) epoch 5775) -2224> 2014-11-08 18:15:35.414998 7fc5b9b29700 5 -- op tracker -- seq: 47260, time: 2014-11-08 18:15:35.414998, event: done, op: pg_notify(4.8fs2(21) epoch 5775) -2223> 2014-11-08 18:15:35.415007 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12093 ==== pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) v5 ==== 12346+0+0 (1287479883 0 0) 0x2127a1c0 con 0x1c5c2680 -2222> 2014-11-08 18:15:35.415015 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:34.574770, event: header_read, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2221> 2014-11-08 18:15:35.415018 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:34.574770, event: throttled, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2220> 2014-11-08 18:15:35.415021 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:34.574985, event: all_read, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2219> 2014-11-08 18:15:35.415024 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:35.415014, event: dispatched, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2218> 2014-11-08 18:15:35.415028 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:35.415028, event: waiting_for_osdmap, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2217> 2014-11-08 18:15:35.415032 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:35.415032, event: started, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2216> 2014-11-08 18:15:35.415062 7fc5b9b29700 5 -- op tracker -- seq: 47261, time: 2014-11-08 18:15:35.415062, event: done, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7) epoch 5775) -2215> 2014-11-08 18:15:35.415095 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 337 ==== pg_notify(4.36s2(40) epoch 5774) v5 ==== 3135+0+0 (150075666 0 0) 0x2234c380 con 0x1c7911e0 -2214> 2014-11-08 18:15:35.415101 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:34.516326, event: header_read, op: pg_notify(4.36s2(40) epoch 5774) -2213> 2014-11-08 18:15:35.415104 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:34.516327, event: throttled, op: pg_notify(4.36s2(40) epoch 5774) -2212> 2014-11-08 18:15:35.415106 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:34.516393, event: all_read, op: pg_notify(4.36s2(40) epoch 5774) -2211> 2014-11-08 18:15:35.415109 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:35.415100, event: dispatched, op: pg_notify(4.36s2(40) epoch 5774) -2210> 2014-11-08 18:15:35.415111 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:35.415111, event: waiting_for_osdmap, op: pg_notify(4.36s2(40) epoch 5774) -2209> 2014-11-08 18:15:35.415114 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:35.415114, event: started, op: pg_notify(4.36s2(40) epoch 5774) -2208> 2014-11-08 18:15:35.415124 7fc5b9b29700 5 -- op tracker -- seq: 47262, time: 2014-11-08 18:15:35.415124, event: done, op: pg_notify(4.36s2(40) epoch 5774) -2207> 2014-11-08 18:15:35.415145 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2299 ==== pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) v5 ==== 18742+0+0 (2692900111 0 0) 0x1d0694a0 con 0x1e50aec0 -2206> 2014-11-08 18:15:35.415153 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.187172, event: header_read, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2205> 2014-11-08 18:15:35.415157 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.187176, event: throttled, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2204> 2014-11-08 18:15:35.415159 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.188319, event: all_read, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2203> 2014-11-08 18:15:35.415162 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.415152, event: dispatched, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2202> 2014-11-08 18:15:35.415165 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.415165, event: waiting_for_osdmap, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2201> 2014-11-08 18:15:35.415169 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.415169, event: started, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2200> 2014-11-08 18:15:35.415191 7fc5b9b29700 5 -- op tracker -- seq: 47263, time: 2014-11-08 18:15:35.415191, event: done, op: pg_notify(4.38s2(53),4.36s2(126),4.21s0(100) epoch 5774) -2199> 2014-11-08 18:15:35.415245 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12094 ==== pg_info(1 pgs e5775:4.24s0(1)) v4 ==== 810+0+0 (222777029 0 0) 0x2127a580 con 0x1c5c2680 -2198> 2014-11-08 18:15:35.415251 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:34.576144, event: header_read, op: pg_info(1 pgs e5775:4.24s0(1)) -2197> 2014-11-08 18:15:35.415254 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:34.576145, event: throttled, op: pg_info(1 pgs e5775:4.24s0(1)) -2196> 2014-11-08 18:15:35.415257 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:34.576187, event: all_read, op: pg_info(1 pgs e5775:4.24s0(1)) -2195> 2014-11-08 18:15:35.415259 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:35.415250, event: dispatched, op: pg_info(1 pgs e5775:4.24s0(1)) -2194> 2014-11-08 18:15:35.415262 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:35.415262, event: waiting_for_osdmap, op: pg_info(1 pgs e5775:4.24s0(1)) -2193> 2014-11-08 18:15:35.415265 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:35.415265, event: started, op: pg_info(1 pgs e5775:4.24s0(1)) -2192> 2014-11-08 18:15:35.415281 7fc5b9b29700 5 -- op tracker -- seq: 47264, time: 2014-11-08 18:15:35.415281, event: done, op: pg_info(1 pgs e5775:4.24s0(1)) -2191> 2014-11-08 18:15:35.415288 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 338 ==== osd_map(5775..5775 src has 4761..5775) v3 ==== 298+0+0 (4097883411 0 0) 0x22acc5c0 con 0x1c7911e0 -2190> 2014-11-08 18:15:35.415296 7fc5b9b29700 3 osd.9 5775 handle_osd_map epochs [5775,5775], i have 5775, src has [4761,5775] -2189> 2014-11-08 18:15:35.415300 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2300 ==== pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) v5 ==== 12138+0+0 (3358000059 0 0) 0x22348000 con 0x1e50aec0 -2188> 2014-11-08 18:15:35.415307 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.188352, event: header_read, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2187> 2014-11-08 18:15:35.415310 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.188353, event: throttled, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2186> 2014-11-08 18:15:35.415313 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.188507, event: all_read, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2185> 2014-11-08 18:15:35.415315 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.415306, event: dispatched, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2184> 2014-11-08 18:15:35.415318 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.415318, event: waiting_for_osdmap, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2183> 2014-11-08 18:15:35.415322 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.415322, event: started, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2182> 2014-11-08 18:15:35.415337 7fc5b9b29700 5 -- op tracker -- seq: 47265, time: 2014-11-08 18:15:35.415337, event: done, op: pg_notify(4.bfs2(127),4.38s2(53) epoch 5774) -2181> 2014-11-08 18:15:35.415378 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12095 ==== pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) v5 ==== 3170+0+0 (2332716671 0 0) 0x1db1f260 con 0x1c5c2680 -2180> 2014-11-08 18:15:35.415404 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:34.580630, event: header_read, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2179> 2014-11-08 18:15:35.415407 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:34.580631, event: throttled, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2178> 2014-11-08 18:15:35.415410 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:34.580693, event: all_read, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2177> 2014-11-08 18:15:35.415412 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:35.415403, event: dispatched, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2176> 2014-11-08 18:15:35.415415 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:35.415415, event: waiting_for_osdmap, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2175> 2014-11-08 18:15:35.415419 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:35.415419, event: started, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2174> 2014-11-08 18:15:35.415436 7fc5b9b29700 5 -- op tracker -- seq: 47266, time: 2014-11-08 18:15:35.415436, event: done, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5775) -2173> 2014-11-08 18:15:35.415453 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 339 ==== pg_notify(4.15s1(28) epoch 5775) v5 ==== 2427+0+0 (4073007957 0 0) 0x22349e00 con 0x1c7911e0 -2172> 2014-11-08 18:15:35.415460 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.014082, event: header_read, op: pg_notify(4.15s1(28) epoch 5775) -2171> 2014-11-08 18:15:35.415462 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.014083, event: throttled, op: pg_notify(4.15s1(28) epoch 5775) -2170> 2014-11-08 18:15:35.415465 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.014160, event: all_read, op: pg_notify(4.15s1(28) epoch 5775) -2169> 2014-11-08 18:15:35.415467 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.415459, event: dispatched, op: pg_notify(4.15s1(28) epoch 5775) -2168> 2014-11-08 18:15:35.415469 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.415469, event: waiting_for_osdmap, op: pg_notify(4.15s1(28) epoch 5775) -2167> 2014-11-08 18:15:35.415473 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.415473, event: started, op: pg_notify(4.15s1(28) epoch 5775) -2166> 2014-11-08 18:15:35.415486 7fc5b9b29700 5 -- op tracker -- seq: 47267, time: 2014-11-08 18:15:35.415486, event: done, op: pg_notify(4.15s1(28) epoch 5775) -2165> 2014-11-08 18:15:35.415504 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2301 ==== MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 v3 ==== 30+0+0 (3434068534 0 0) 0x239e9fe0 con 0x1e50aec0 -2164> 2014-11-08 18:15:35.415512 7fc5b9b29700 5 -- op tracker -- seq: 47268, time: 2014-11-08 18:15:35.188609, event: header_read, op: MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 -2163> 2014-11-08 18:15:35.415514 7fc5b9b29700 5 -- op tracker -- seq: 47268, time: 2014-11-08 18:15:35.188611, event: throttled, op: MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 -2162> 2014-11-08 18:15:35.415517 7fc5b9b29700 5 -- op tracker -- seq: 47268, time: 2014-11-08 18:15:35.188670, event: all_read, op: MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 -2161> 2014-11-08 18:15:35.415519 7fc5b9b29700 5 -- op tracker -- seq: 47268, time: 2014-11-08 18:15:35.415511, event: dispatched, op: MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 -2160> 2014-11-08 18:15:35.415522 7fc5b9b29700 5 -- op tracker -- seq: 47268, time: 2014-11-08 18:15:35.415522, event: waiting_for_osdmap, op: MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 -2159> 2014-11-08 18:15:35.415530 7fc5b9b29700 5 -- op tracker -- seq: 47268, time: 2014-11-08 18:15:35.415530, event: done, op: MBackfillReserve REQUEST pgid: 4.2fs2, query_epoch: 5774, prio: 201 -2158> 2014-11-08 18:15:35.415537 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12096 ==== pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) v4 ==== 2414+0+0 (1225613424 0 0) 0x2286d820 con 0x1c5c2680 -2157> 2014-11-08 18:15:35.415651 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:34.580923, event: header_read, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2156> 2014-11-08 18:15:35.415655 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:34.580923, event: throttled, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2155> 2014-11-08 18:15:35.415658 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:34.580968, event: all_read, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2154> 2014-11-08 18:15:35.415661 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:35.415650, event: dispatched, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2153> 2014-11-08 18:15:35.415664 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:35.415664, event: waiting_for_osdmap, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2152> 2014-11-08 18:15:35.415668 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:35.415668, event: started, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2151> 2014-11-08 18:15:35.415701 7fc5b9b29700 5 -- op tracker -- seq: 47269, time: 2014-11-08 18:15:35.415701, event: done, op: pg_info(3 pgs e5775:4.99s0(1),4.9fs0(1),4.8ds0(1)) -2150> 2014-11-08 18:15:35.415709 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 340 ==== pg_notify(4.36s2(40) epoch 5775) v5 ==== 3135+0+0 (2501187228 0 0) 0x22455fa0 con 0x1c7911e0 -2149> 2014-11-08 18:15:35.415716 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.031395, event: header_read, op: pg_notify(4.36s2(40) epoch 5775) -2148> 2014-11-08 18:15:35.415719 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.031398, event: throttled, op: pg_notify(4.36s2(40) epoch 5775) -2147> 2014-11-08 18:15:35.415721 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.031514, event: all_read, op: pg_notify(4.36s2(40) epoch 5775) -2146> 2014-11-08 18:15:35.415724 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.415715, event: dispatched, op: pg_notify(4.36s2(40) epoch 5775) -2145> 2014-11-08 18:15:35.415726 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.415726, event: waiting_for_osdmap, op: pg_notify(4.36s2(40) epoch 5775) -2144> 2014-11-08 18:15:35.415729 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.415729, event: started, op: pg_notify(4.36s2(40) epoch 5775) -2143> 2014-11-08 18:15:35.415740 7fc5b9b29700 5 -- op tracker -- seq: 47270, time: 2014-11-08 18:15:35.415740, event: done, op: pg_notify(4.36s2(40) epoch 5775) -2142> 2014-11-08 18:15:35.415752 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2302 ==== pg_notify(4.36s2(126) epoch 5774) v5 ==== 8189+0+0 (3800644537 0 0) 0x1eaccce0 con 0x1e50aec0 -2141> 2014-11-08 18:15:35.415758 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.188730, event: header_read, op: pg_notify(4.36s2(126) epoch 5774) -2140> 2014-11-08 18:15:35.415761 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.188731, event: throttled, op: pg_notify(4.36s2(126) epoch 5774) -2139> 2014-11-08 18:15:35.415763 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.188833, event: all_read, op: pg_notify(4.36s2(126) epoch 5774) -2138> 2014-11-08 18:15:35.415766 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.415758, event: dispatched, op: pg_notify(4.36s2(126) epoch 5774) -2137> 2014-11-08 18:15:35.415768 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.415768, event: waiting_for_osdmap, op: pg_notify(4.36s2(126) epoch 5774) -2136> 2014-11-08 18:15:35.415772 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.415772, event: started, op: pg_notify(4.36s2(126) epoch 5774) -2135> 2014-11-08 18:15:35.415781 7fc5b9b29700 5 -- op tracker -- seq: 47271, time: 2014-11-08 18:15:35.415781, event: done, op: pg_notify(4.36s2(126) epoch 5774) -2134> 2014-11-08 18:15:35.415805 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12097 ==== pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) v5 ==== 8672+0+0 (171765982 0 0) 0x1dbfb2a0 con 0x1c5c2680 -2133> 2014-11-08 18:15:35.415812 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:34.590053, event: header_read, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2132> 2014-11-08 18:15:35.415815 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:34.590055, event: throttled, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2131> 2014-11-08 18:15:35.415818 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:34.592741, event: all_read, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2130> 2014-11-08 18:15:35.415821 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:35.415811, event: dispatched, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2129> 2014-11-08 18:15:35.415825 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:35.415824, event: waiting_for_osdmap, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2128> 2014-11-08 18:15:35.415828 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:35.415828, event: started, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2127> 2014-11-08 18:15:35.415856 7fc5b9b29700 5 -- op tracker -- seq: 47272, time: 2014-11-08 18:15:35.415856, event: done, op: pg_notify(4.cs1(34),4.es2(34),4.bfs0(41) epoch 5775) -2126> 2014-11-08 18:15:35.415883 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12098 ==== pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) v4 ==== 2084+0+0 (2528145485 0 0) 0x1d1b0d20 con 0x1c5c2680 -2125> 2014-11-08 18:15:35.415891 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:34.592751, event: header_read, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2124> 2014-11-08 18:15:35.415894 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:34.592752, event: throttled, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2123> 2014-11-08 18:15:35.415896 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:34.592804, event: all_read, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2122> 2014-11-08 18:15:35.415899 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:35.415890, event: dispatched, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2121> 2014-11-08 18:15:35.415902 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:35.415902, event: waiting_for_osdmap, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2120> 2014-11-08 18:15:35.415905 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:35.415905, event: started, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2119> 2014-11-08 18:15:35.415921 7fc5b9b29700 5 -- op tracker -- seq: 47273, time: 2014-11-08 18:15:35.415921, event: done, op: pg_info(2 pgs e5775:4.ds0(1),4.7s1(9)) -2118> 2014-11-08 18:15:35.415929 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12099 ==== pg_notify(4.75s1(7) epoch 5775) v5 ==== 1168+0+0 (3943017997 0 0) 0x1d1b41a0 con 0x1c5c2680 -2117> 2014-11-08 18:15:35.415936 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:34.604856, event: header_read, op: pg_notify(4.75s1(7) epoch 5775) -2116> 2014-11-08 18:15:35.415938 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:34.604860, event: throttled, op: pg_notify(4.75s1(7) epoch 5775) -2115> 2014-11-08 18:15:35.415941 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:34.604965, event: all_read, op: pg_notify(4.75s1(7) epoch 5775) -2114> 2014-11-08 18:15:35.415943 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:35.415935, event: dispatched, op: pg_notify(4.75s1(7) epoch 5775) -2113> 2014-11-08 18:15:35.415945 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:35.415945, event: waiting_for_osdmap, op: pg_notify(4.75s1(7) epoch 5775) -2112> 2014-11-08 18:15:35.415949 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:35.415949, event: started, op: pg_notify(4.75s1(7) epoch 5775) -2111> 2014-11-08 18:15:35.415959 7fc5b9b29700 5 -- op tracker -- seq: 47274, time: 2014-11-08 18:15:35.415959, event: done, op: pg_notify(4.75s1(7) epoch 5775) -2110> 2014-11-08 18:15:35.415967 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12100 ==== pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) v4 ==== 8974+0+0 (3716863058 0 0) 0x25ed5280 con 0x1c5c2680 -2109> 2014-11-08 18:15:35.415975 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:34.604982, event: header_read, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2108> 2014-11-08 18:15:35.415979 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:34.604982, event: throttled, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2107> 2014-11-08 18:15:35.415982 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:34.607179, event: all_read, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2106> 2014-11-08 18:15:35.415986 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:35.415974, event: dispatched, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2105> 2014-11-08 18:15:35.415989 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:35.415989, event: waiting_for_osdmap, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2104> 2014-11-08 18:15:35.415994 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:35.415994, event: started, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2103> 2014-11-08 18:15:35.416027 7fc5b9b29700 5 -- op tracker -- seq: 47275, time: 2014-11-08 18:15:35.416027, event: done, op: pg_info(5 pgs e5775:4.6ds0(1),4.67s1(32),4.67s1(32),4.5as0(11),4.59s1(13)) -2102> 2014-11-08 18:15:35.416047 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12101 ==== pg_notify(4.31s2(22),4.36s0(40) epoch 5775) v5 ==== 5156+0+0 (3410661209 0 0) 0x25ed4ec0 con 0x1c5c2680 -2101> 2014-11-08 18:15:35.416054 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:34.607191, event: header_read, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2100> 2014-11-08 18:15:35.416057 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:34.607192, event: throttled, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2099> 2014-11-08 18:15:35.416059 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:34.607292, event: all_read, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2098> 2014-11-08 18:15:35.416062 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:35.416053, event: dispatched, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2097> 2014-11-08 18:15:35.416065 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:35.416064, event: waiting_for_osdmap, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2096> 2014-11-08 18:15:35.416068 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:35.416068, event: started, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2095> 2014-11-08 18:15:35.416087 7fc5b9b29700 5 -- op tracker -- seq: 47276, time: 2014-11-08 18:15:35.416086, event: done, op: pg_notify(4.31s2(22),4.36s0(40) epoch 5775) -2094> 2014-11-08 18:15:35.416257 7fc5c362f700 5 osd.9 5775 tick -2093> 2014-11-08 18:15:35.416103 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12102 ==== pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) v5 ==== 8904+0+0 (2483294121 0 0) 0x25ed6900 con 0x1c5c2680 -2092> 2014-11-08 18:15:35.420220 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:34.609314, event: header_read, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2091> 2014-11-08 18:15:35.420232 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:34.609318, event: throttled, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2090> 2014-11-08 18:15:35.420236 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:34.609661, event: all_read, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2089> 2014-11-08 18:15:35.420239 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:35.420215, event: dispatched, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2088> 2014-11-08 18:15:35.420243 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:35.420243, event: waiting_for_osdmap, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2087> 2014-11-08 18:15:35.420250 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:35.420250, event: started, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2086> 2014-11-08 18:15:35.420290 7fc5b9b29700 5 -- op tracker -- seq: 47277, time: 2014-11-08 18:15:35.420290, event: done, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30),4.39s1(7) epoch 5775) -2085> 2014-11-08 18:15:35.420428 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12103 ==== pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) v4 ==== 6450+0+0 (4273944924 0 0) 0x258390e0 con 0x1c5c2680 -2084> 2014-11-08 18:15:35.420438 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:34.609676, event: header_read, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2083> 2014-11-08 18:15:35.420441 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:34.609677, event: throttled, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2082> 2014-11-08 18:15:35.420444 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:34.609898, event: all_read, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2081> 2014-11-08 18:15:35.420446 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:35.420437, event: dispatched, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2080> 2014-11-08 18:15:35.420449 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:35.420449, event: waiting_for_osdmap, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2079> 2014-11-08 18:15:35.420453 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:35.420453, event: started, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2078> 2014-11-08 18:15:35.420473 7fc5b9b29700 5 -- op tracker -- seq: 47278, time: 2014-11-08 18:15:35.420473, event: done, op: pg_info(2 pgs e5775:4.57s0(11),4.45s1(73)) -2077> 2014-11-08 18:15:35.421042 7fc5adb11700 5 -- op tracker -- seq: 47254, time: 2014-11-08 18:15:35.421042, event: done, op: MOSDECSubOpReadReply(4.83s0 5774 ECSubReadReply(tid=4427, attrs_read=0)) -2076> 2014-11-08 18:15:35.422655 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 1.255983 8 0.002200 -2075> 2014-11-08 18:15:35.422697 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -2074> 2014-11-08 18:15:35.422763 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000066 0 0.000000 -2073> 2014-11-08 18:15:35.422775 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -2072> 2014-11-08 18:15:35.422789 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000014 0 0.000000 -2071> 2014-11-08 18:15:35.422799 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -2070> 2014-11-08 18:15:35.422876 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetInfo 1.229147 10 0.000147 -2069> 2014-11-08 18:15:35.422887 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetLog -2068> 2014-11-08 18:15:35.422924 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetLog 0.000037 0 0.000000 -2067> 2014-11-08 18:15:35.422935 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/GetMissing -2066> 2014-11-08 18:15:35.422946 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/GetMissing 0.000011 0 0.000000 -2065> 2014-11-08 18:15:35.422956 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] enter Started/Primary/Peering/WaitUpThru -2064> 2014-11-08 18:15:35.422977 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] exit Started/Stray 1.255161 7 0.000078 -2063> 2014-11-08 18:15:35.422989 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] enter Started/ReplicaActive -2062> 2014-11-08 18:15:35.423011 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.24s1( v 5734'8095 (3458'5066,5734'8095] local-les=5753 n=7616 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5721'8088 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2061> 2014-11-08 18:15:35.423279 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.2fs2( v 5734'8483 (3458'5435,5734'8483] lb b8d0202f/1000002b572.00000031/head//4 local-les=0 n=58 ec=195 les/c 5760/4864 5759/5759/5759) [5,1,9]/[2147483647,5,2] r=-1 lpr=5759 pi=4863-5758/42 luod=0'0 crt=5734'8483 active+remapped] exit Started/ReplicaActive/RepNotRecovering 1010.096895 39 0.000524 -2060> 2014-11-08 18:15:35.423292 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.2fs2( v 5734'8483 (3458'5435,5734'8483] lb b8d0202f/1000002b572.00000031/head//4 local-les=0 n=58 ec=195 les/c 5760/4864 5759/5759/5759) [5,1,9]/[2147483647,5,2] r=-1 lpr=5759 pi=4863-5758/42 luod=0'0 crt=5734'8483 active+remapped] enter Started/ReplicaActive/RepWaitBackfillReserved -2059> 2014-11-08 18:15:35.423306 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] exit Started/Stray 1.251284 7 0.000220 -2058> 2014-11-08 18:15:35.423315 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] enter Started/ReplicaActive -2057> 2014-11-08 18:15:35.423322 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.99s1( v 5734'8086 (3458'5085,5734'8086] local-les=5753 n=7592 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8075 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2056> 2014-11-08 18:15:35.423539 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] exit Started/Stray 1.243869 7 0.006241 -2055> 2014-11-08 18:15:35.423549 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] enter Started/ReplicaActive -2054> 2014-11-08 18:15:35.423556 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.9fs2( v 5734'8365 (3458'5364,5734'8365] local-les=5753 n=7732 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2951-5773/42 crt=5734'8359 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2053> 2014-11-08 18:15:35.424248 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] exit Started/Stray 1.236090 7 0.000357 -2052> 2014-11-08 18:15:35.424295 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] enter Started/ReplicaActive -2051> 2014-11-08 18:15:35.424305 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.5as2( v 5734'16392 (3473'13323,5734'16392] local-les=5753 n=15330 ec=195 les/c 5753/5525 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=2790-5773/93 crt=5734'16383 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2050> 2014-11-08 18:15:35.424643 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] exit Started/Stray 1.234803 7 0.000663 -2049> 2014-11-08 18:15:35.424657 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] enter Started/ReplicaActive -2048> 2014-11-08 18:15:35.424665 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.57s1( v 5734'16439 (3473'13379,5734'16439] local-les=5753 n=15365 ec=195 les/c 5753/5525 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=4252-5773/69 crt=5734'16425 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2047> 2014-11-08 18:15:35.425097 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] exit Started/Stray 1.233553 7 0.000140 -2046> 2014-11-08 18:15:35.425110 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] enter Started/ReplicaActive -2045> 2014-11-08 18:15:35.425118 7fc5b1b19700 5 osd.9 pg_epoch: 5775 pg[4.45s2( v 5734'16249 (3473'13248,5734'16249] local-les=5752 n=15193 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16225 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2044> 2014-11-08 18:15:35.425839 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] exit Started/Stray 1.252338 7 0.000776 -2043> 2014-11-08 18:15:35.425860 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] enter Started/ReplicaActive -2042> 2014-11-08 18:15:35.425876 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.8ds1( v 5734'8111 (3458'5110,5734'8111] local-les=5753 n=7583 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8105 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2041> 2014-11-08 18:15:35.426184 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] exit Started/Stray 1.260468 7 0.000107 -2040> 2014-11-08 18:15:35.426299 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] enter Started/ReplicaActive -2039> 2014-11-08 18:15:35.426316 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.ds1( v 5734'8170 (3458'5169,5734'8170] local-les=5753 n=7676 ec=195 les/c 5753/5756 5774/5774/5641) [1,9,2147483647] r=1 lpr=5774 pi=5635-5773/11 crt=5734'8164 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2038> 2014-11-08 18:15:35.426692 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] exit Started/Stray 1.260531 7 0.000150 -2037> 2014-11-08 18:15:35.426711 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] enter Started/ReplicaActive -2036> 2014-11-08 18:15:35.426725 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.7s2( v 5734'8166 (3458'5165,5734'8166] local-les=5752 n=7608 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=5432-5773/24 crt=5734'8162 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2035> 2014-11-08 18:15:35.427125 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] exit Started/Stray 1.162778 7 0.000146 -2034> 2014-11-08 18:15:35.427145 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] enter Started/ReplicaActive -2033> 2014-11-08 18:15:35.427160 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.6ds2( v 5734'16396 (3473'13395,5734'16396] local-les=5753 n=15394 ec=195 les/c 5753/5756 5774/5774/5641) [1,2147483647,9] r=2 lpr=5774 pi=4863-5773/34 crt=5734'16391 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2032> 2014-11-08 18:15:35.427707 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3688 ==== osd_ping(ping e5774 stamp 2014-11-08 18:15:35.425324) v2 ==== 47+0+0 (1715335396 0 0) 0x1caaba20 con 0x1cce2ec0 -2031> 2014-11-08 18:15:35.427879 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3688 ==== osd_ping(ping e5774 stamp 2014-11-08 18:15:35.425324) v2 ==== 47+0+0 (1715335396 0 0) 0x23683fc0 con 0x1cce3440 -2030> 2014-11-08 18:15:35.427898 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5775 stamp 2014-11-08 18:15:35.425324) v2 -- ?+0 0x1e6ab840 con 0x1cce2ec0 -2029> 2014-11-08 18:15:35.427928 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5775 stamp 2014-11-08 18:15:35.425324) v2 -- ?+0 0x2127d640 con 0x1cce3440 -2028> 2014-11-08 18:15:35.428248 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] exit Started/Stray 1.163400 7 0.000100 -2027> 2014-11-08 18:15:35.428293 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] enter Started/ReplicaActive -2026> 2014-11-08 18:15:35.428312 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.67s1( v 5734'16521 (3473'13445,5734'16521] lb c90ff67/1000001b40e.00000016/head//4 local-les=0 n=15673 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=-1 lpr=5774 pi=4872-5773/29 crt=5734'16521 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2025> 2014-11-08 18:15:35.428659 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] exit Started/Stray 1.163624 7 0.000158 -2024> 2014-11-08 18:15:35.428678 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] enter Started/ReplicaActive -2023> 2014-11-08 18:15:35.428702 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.67s2( v 5734'16521 (3473'13444,5734'16521] local-les=5753 n=15518 ec=195 les/c 5753/4877 5774/5774/5641) [1,9,2147483647]/[2147483647,1,9] r=2 lpr=5774 pi=2951-5773/78 crt=5734'16515 remapped NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2022> 2014-11-08 18:15:35.429139 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] exit Started/Stray 1.242884 7 0.000252 -2021> 2014-11-08 18:15:35.429158 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] enter Started/ReplicaActive -2020> 2014-11-08 18:15:35.429173 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.59s2( v 5734'16563 (3473'13562,5734'16563] local-les=5752 n=15459 ec=195 les/c 5752/5756 5774/5774/5774) [2147483647,1,9] r=2 lpr=5774 pi=2936-5773/61 crt=5734'16540 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering -2019> 2014-11-08 18:15:35.429610 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- MBackfillReserve GRANT pgid: 4.2fs1, query_epoch: 5775 v3 -- ?+0 0x21278b40 con 0x1e50aec0 -2018> 2014-11-08 18:15:35.429628 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.2fs2( v 5734'8483 (3458'5435,5734'8483] lb b8d0202f/1000002b572.00000031/head//4 local-les=0 n=58 ec=195 les/c 5760/4864 5759/5759/5759) [5,1,9]/[2147483647,5,2] r=-1 lpr=5759 pi=4863-5758/42 luod=0'0 crt=5734'8483 active+remapped] exit Started/ReplicaActive/RepWaitBackfillReserved 0.006336 1 0.000032 -2017> 2014-11-08 18:15:35.429650 7fc5b1318700 5 osd.9 pg_epoch: 5775 pg[4.2fs2( v 5734'8483 (3458'5435,5734'8483] lb b8d0202f/1000002b572.00000031/head//4 local-les=0 n=58 ec=195 les/c 5760/4864 5759/5759/5759) [5,1,9]/[2147483647,5,2] r=-1 lpr=5759 pi=4863-5758/42 luod=0'0 crt=5734'8483 active+remapped] enter Started/ReplicaActive/RepRecovering -2016> 2014-11-08 18:15:35.463450 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.1 192.168.1.15:0/1411 3722 ==== osd_ping(ping e5775 stamp 2014-11-08 18:15:35.459081) v2 ==== 47+0+0 (725501495 0 0) 0x257c1e00 con 0x1c9f8580 -2015> 2014-11-08 18:15:35.463560 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.1 192.168.1.15:0/1411 3722 ==== osd_ping(ping e5775 stamp 2014-11-08 18:15:35.459081) v2 ==== 47+0+0 (725501495 0 0) 0x22d50f00 con 0x1c8b7d20 -2014> 2014-11-08 18:15:35.464136 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5775 stamp 2014-11-08 18:15:35.459081) v2 -- ?+0 0x1caaba20 con 0x1c9f8580 -2013> 2014-11-08 18:15:35.464191 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1411 -- osd_ping(ping_reply e5775 stamp 2014-11-08 18:15:35.459081) v2 -- ?+0 0x23683fc0 con 0x1c8b7d20 -2012> 2014-11-08 18:15:35.478335 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1511 ==== osd_map(5775..5776 src has 4761..5776) v3 ==== 504+0+0 (3965797669 0 0) 0x22ac8900 con 0x44ed6a0 -2011> 2014-11-08 18:15:35.478378 7fc5bb32c700 3 osd.9 5775 handle_osd_map epochs [5775,5776], i have 5775, src has [4761,5776] -2010> 2014-11-08 18:15:35.479347 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.285097 7 0.000093 -2009> 2014-11-08 18:15:35.479368 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 peering] exit Started/Primary/Peering 1.328362 0 0.000000 -2008> 2014-11-08 18:15:35.479380 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5752 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 inactive] enter Started/Primary/Active -2007> 2014-11-08 18:15:35.479573 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.28s0( v 5734'8247 (3458'5246,5734'8247] local-les=5776 n=7693 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8245 lcod 5734'8246 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -2006> 2014-11-08 18:15:35.481361 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.285660 7 0.000095 -2005> 2014-11-08 18:15:35.481389 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 peering] exit Started/Primary/Peering 1.330301 0 0.000000 -2004> 2014-11-08 18:15:35.481401 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5752 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 inactive] enter Started/Primary/Active -2003> 2014-11-08 18:15:35.483346 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.as0( v 5734'8327 (3458'5326,5734'8327] local-les=5776 n=7800 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8320 lcod 5734'8326 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -2002> 2014-11-08 18:15:35.483485 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.289551 8 0.000101 -2001> 2014-11-08 18:15:35.483497 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 peering] exit Started/Primary/Peering 1.318131 0 0.000000 -2000> 2014-11-08 18:15:35.483508 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5752 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 inactive] enter Started/Primary/Active -1999> 2014-11-08 18:15:35.483708 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5776 n=7653 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1998> 2014-11-08 18:15:35.483758 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.289671 8 0.000092 -1997> 2014-11-08 18:15:35.483768 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 peering] exit Started/Primary/Peering 1.318237 0 0.000000 -1996> 2014-11-08 18:15:35.483777 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5752 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 inactive] enter Started/Primary/Active -1995> 2014-11-08 18:15:35.483969 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5776 n=7729 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1994> 2014-11-08 18:15:35.484041 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.290035 8 0.000078 -1993> 2014-11-08 18:15:35.484051 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 peering] exit Started/Primary/Peering 1.318103 0 0.000000 -1992> 2014-11-08 18:15:35.484061 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5756 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 inactive] enter Started/Primary/Active -1991> 2014-11-08 18:15:35.484300 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5776 n=7607 ec=195 les/c 5756/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1990> 2014-11-08 18:15:35.484365 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.290018 7 0.000086 -1989> 2014-11-08 18:15:35.484376 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 peering] exit Started/Primary/Peering 1.318046 0 0.000000 -1988> 2014-11-08 18:15:35.484386 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5752 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 inactive] enter Started/Primary/Active -1987> 2014-11-08 18:15:35.488153 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 1.290851 12 0.000124 -1986> 2014-11-08 18:15:35.488182 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.319573 0 0.000000 -1985> 2014-11-08 18:15:35.488196 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5752 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 remapped] enter Started/Primary/Active -1984> 2014-11-08 18:15:35.488422 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.15s1( v 5734'8279 (3458'5251,5734'8279] local-les=5776 n=7670 ec=195 les/c 5752/4877 5774/5774/5728) [1,10,2147483647]/[2147483647,9,1] r=1 lpr=5774 pi=4860-5773/31 bft=1(0),10(1) crt=5734'8272 lcod 5734'8278 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -1983> 2014-11-08 18:15:35.488510 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- osd_map(5776..5776 src has 4761..5776) v3 -- ?+0 0x22acc140 con 0x1c5c2680 -1982> 2014-11-08 18:15:35.488526 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_notify(4.12s1(53) epoch 5776) v5 -- ?+0 0x234e32a0 con 0x1c5c2680 -1981> 2014-11-08 18:15:35.488535 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- osd_map(5776..5776 src has 4761..5776) v3 -- ?+0 0x22acbf00 con 0x1c5c0580 -1980> 2014-11-08 18:15:35.488543 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.2as2(35),4.2ds2(46),4.14s2(44) epoch 5776) v5 -- ?+0 0x234e1680 con 0x1c5c0580 -1979> 2014-11-08 18:15:35.488556 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.21s2(50) epoch 5776) v5 -- ?+0 0x234e1a40 con 0x42c86e0 -1978> 2014-11-08 18:15:35.488731 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.21s2(50) epoch 5776) v5 ==== 0+0+0 (0 0 0) 0x234e1a40 con 0x42c86e0 -1977> 2014-11-08 18:15:35.488747 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 2014-11-08 18:15:35.488561, event: header_read, op: pg_notify(4.21s2(50) epoch 5776) -1976> 2014-11-08 18:15:35.488755 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 0.000000, event: throttled, op: pg_notify(4.21s2(50) epoch 5776) -1975> 2014-11-08 18:15:35.488757 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 0.000000, event: all_read, op: pg_notify(4.21s2(50) epoch 5776) -1974> 2014-11-08 18:15:35.488759 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 2014-11-08 18:15:35.488742, event: dispatched, op: pg_notify(4.21s2(50) epoch 5776) -1973> 2014-11-08 18:15:35.488763 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 2014-11-08 18:15:35.488763, event: waiting_for_osdmap, op: pg_notify(4.21s2(50) epoch 5776) -1972> 2014-11-08 18:15:35.488770 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 2014-11-08 18:15:35.488770, event: started, op: pg_notify(4.21s2(50) epoch 5776) -1971> 2014-11-08 18:15:35.488787 7fc5b9b29700 5 -- op tracker -- seq: 47279, time: 2014-11-08 18:15:35.488787, event: done, op: pg_notify(4.21s2(50) epoch 5776) -1970> 2014-11-08 18:15:35.489232 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(2 pgs e5776:4.15s1(31),4.15s1(31)) v4 -- ?+0 0x234e1fe0 con 0x1c5c2680 -1969> 2014-11-08 18:15:35.489246 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_info(1 pgs e5776:4.28s0(1)) v4 -- ?+0 0x234e7bc0 con 0x1c5c0580 -1968> 2014-11-08 18:15:35.489257 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- osd_map(5776..5776 src has 4761..5776) v3 -- ?+0 0x22acbcc0 con 0x1c7911e0 -1967> 2014-11-08 18:15:35.489272 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_info(1 pgs e5776:4.15s1(31)) v4 -- ?+0 0x234e4380 con 0x1c7911e0 -1966> 2014-11-08 18:15:35.489448 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.294330 7 0.000097 -1965> 2014-11-08 18:15:35.489462 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 peering] exit Started/Primary/Peering 1.318425 0 0.000000 -1964> 2014-11-08 18:15:35.489473 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5752 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 inactive] enter Started/Primary/Active -1963> 2014-11-08 18:15:35.489822 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1512 ==== osd_map(5775..5776 src has 4761..5776) v3 ==== 504+0+0 (3965797669 0 0) 0x22ac86c0 con 0x44ed6a0 -1962> 2014-11-08 18:15:35.489841 7fc5bb32c700 3 osd.9 5776 handle_osd_map epochs [5775,5776], i have 5776, src has [4761,5776] -1961> 2014-11-08 18:15:35.489934 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.5s0( v 5734'8299 (3458'5298,5734'8299] local-les=5776 n=7704 ec=195 les/c 5752/5752 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8293 lcod 5734'8298 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1960> 2014-11-08 18:15:35.490077 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.067278 4 0.000170 -1959> 2014-11-08 18:15:35.490091 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.323425 0 0.000000 -1958> 2014-11-08 18:15:35.490103 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5765 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 remapped] enter Started/Primary/Active -1957> 2014-11-08 18:15:35.491394 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.bfs1( v 5734'8212 (3458'5184,5734'8212] local-les=5776 n=7726 ec=195 les/c 5765/5454 5774/5774/5728) [2147483647,9,2]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/41 bft=2(2) crt=5734'8206 lcod 5734'8211 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -1956> 2014-11-08 18:15:35.491591 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.297155 7 0.000089 -1955> 2014-11-08 18:15:35.491603 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 peering] exit Started/Primary/Peering 1.322166 0 0.000000 -1954> 2014-11-08 18:15:35.491613 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5752 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 inactive] enter Started/Primary/Active -1953> 2014-11-08 18:15:35.491834 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.b3s0( v 5734'8164 (3458'5163,5734'8164] local-les=5776 n=7654 ec=195 les/c 5752/5752 5774/5774/5729) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8162 lcod 5734'8163 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1952> 2014-11-08 18:15:35.489684 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.a0s0( v 5734'8326 (3458'5325,5734'8326] local-les=5776 n=7747 ec=195 les/c 5752/5753 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8319 lcod 5734'8325 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1951> 2014-11-08 18:15:35.492074 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.6s2(24),4.bcs1(76),4.bds1(19),4.b2s2(71),4.b5s1(22) epoch 5776) v5 -- ?+0 0x21278960 con 0x1c5c0580 -1950> 2014-11-08 18:15:35.492097 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(3 pgs e5776:4.9s0(1),4.es0(1),4.cs0(14)) v4 -- ?+0 0x21278780 con 0x1c5c2680 -1949> 2014-11-08 18:15:35.492112 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_info(4 pgs e5776:4.as0(1),4.5s0(1),4.bfs1(41),4.b3s0(1)) v4 -- ?+0 0x212785a0 con 0x1c5c0580 -1948> 2014-11-08 18:15:35.492127 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- osd_map(5776..5776 src has 4761..5776) v3 -- ?+0 0x224a9f80 con 0x1e50aec0 -1947> 2014-11-08 18:15:35.492141 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_info(1 pgs e5776:4.bfs1(41)) v4 -- ?+0 0x212783c0 con 0x1e50aec0 -1946> 2014-11-08 18:15:35.492160 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.296944 7 0.000090 -1945> 2014-11-08 18:15:35.492172 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 peering] exit Started/Primary/Peering 1.320614 0 0.000000 -1944> 2014-11-08 18:15:35.492221 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5752 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 inactive] enter Started/Primary/Active -1943> 2014-11-08 18:15:35.492260 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.225927 8 0.001077 -1942> 2014-11-08 18:15:35.492273 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 peering] exit Started/Primary/Peering 1.228836 0 0.000000 -1941> 2014-11-08 18:15:35.492283 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5752 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 inactive] enter Started/Primary/Active -1940> 2014-11-08 18:15:35.492414 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.98s0( v 5734'8295 (3458'5294,5734'8295] local-les=5776 n=7719 ec=195 les/c 5752/5752 5774/5774/5731) [9,2147483647,2] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8285 lcod 5734'8294 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1939> 2014-11-08 18:15:35.492475 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5776 n=7680 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1938> 2014-11-08 18:15:35.492529 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.297218 7 0.000088 -1937> 2014-11-08 18:15:35.492539 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 peering] exit Started/Primary/Peering 1.312020 0 0.000000 -1936> 2014-11-08 18:15:35.492556 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5752 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 inactive] enter Started/Primary/Active -1935> 2014-11-08 18:15:35.492986 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.226531 8 0.000116 -1934> 2014-11-08 18:15:35.492999 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 peering] exit Started/Primary/Peering 1.228853 0 0.000000 -1933> 2014-11-08 18:15:35.493009 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5752 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 inactive] enter Started/Primary/Active -1932> 2014-11-08 18:15:35.493337 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5776 n=15343 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1931> 2014-11-08 18:15:35.493779 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.226979 7 0.000076 -1930> 2014-11-08 18:15:35.493792 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 peering] exit Started/Primary/Peering 1.228583 0 0.000000 -1929> 2014-11-08 18:15:35.493802 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5752 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 inactive] enter Started/Primary/Active -1928> 2014-11-08 18:15:35.494067 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.65s1( v 5734'16319 (3473'13318,5734'16319] local-les=5776 n=15374 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'16313 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1927> 2014-11-08 18:15:35.494106 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.86s2(77),4.7cs2(48),4.61s2(74) epoch 5776) v5 -- ?+0 0x1e5321c0 con 0x1c5c0580 -1926> 2014-11-08 18:15:35.494118 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(2 pgs e5776:4.81s0(1),4.75s0(1)) v4 -- ?+0 0x1e536ae0 con 0x1c5c2680 -1925> 2014-11-08 18:15:35.494125 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_info(1 pgs e5776:4.65s1(2)) v4 -- ?+0 0x1e5310e0 con 0x1c5c0580 -1924> 2014-11-08 18:15:35.497822 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.9ds1( v 5734'8321 (3458'5320,5734'8321] local-les=5776 n=7759 ec=195 les/c 5752/5755 5774/5774/5774) [2147483647,9,2] r=1 lpr=5774 pi=5749-5773/2 crt=5734'8302 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1923> 2014-11-08 18:15:35.497817 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.301391 8 0.000095 -1922> 2014-11-08 18:15:35.497867 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 peering] exit Started/Primary/Peering 1.306490 0 0.000000 -1921> 2014-11-08 18:15:35.497882 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5769 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 inactive] enter Started/Primary/Active -1920> 2014-11-08 18:15:35.497937 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.302543 7 0.000088 -1919> 2014-11-08 18:15:35.497956 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 peering] exit Started/Primary/Peering 1.316523 0 0.000000 -1918> 2014-11-08 18:15:35.497974 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5752 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 inactive] enter Started/Primary/Active -1917> 2014-11-08 18:15:35.498096 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5776 n=15308 ec=195 les/c 5769/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1916> 2014-11-08 18:15:35.498166 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.301656 8 0.000088 -1915> 2014-11-08 18:15:35.498178 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 peering] exit Started/Primary/Peering 1.306298 0 0.000000 -1914> 2014-11-08 18:15:35.498189 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5752 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 inactive] enter Started/Primary/Active -1913> 2014-11-08 18:15:35.498962 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5776 n=7739 ec=195 les/c 5752/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1912> 2014-11-08 18:15:35.499038 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.5bs1(11),4.49s2(54) epoch 5776) v5 -- ?+0 0x227cbc00 con 0x1c5c0580 -1911> 2014-11-08 18:15:35.499054 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.5ds0(91) epoch 5776) v5 -- ?+0 0x227c9a40 con 0x1c7911e0 -1910> 2014-11-08 18:15:35.499065 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(2 pgs e5776:4.40s0(1),4.39s0(1)) v4 -- ?+0 0x213beae0 con 0x1c5c2680 -1909> 2014-11-08 18:15:35.499261 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] exit Started/Primary/Peering/WaitUpThru 1.302666 8 0.000094 -1908> 2014-11-08 18:15:35.499274 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 peering] exit Started/Primary/Peering 1.305815 0 0.000000 -1907> 2014-11-08 18:15:35.499284 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5752 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 inactive] enter Started/Primary/Active -1906> 2014-11-08 18:15:35.499474 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5776 n=7614 ec=195 les/c 5752/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1905> 2014-11-08 18:15:35.499526 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering/WaitUpThru 0.076571 5 0.000103 -1904> 2014-11-08 18:15:35.499538 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped+peering] exit Started/Primary/Peering 1.305816 0 0.000000 -1903> 2014-11-08 18:15:35.499550 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5765 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 remapped] enter Started/Primary/Active -1902> 2014-11-08 18:15:35.500127 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.36s1( v 5734'8279 (3458'5231,5734'8279] local-les=5776 n=7731 ec=195 les/c 5765/5454 5774/5774/5728) [1,2147483647,10]/[2147483647,9,5] r=1 lpr=5774 pi=5453-5773/40 bft=1(0),10(2) crt=5734'8263 lcod 5734'8278 mlcod 0'0 undersized+degraded+remapped] enter Started/Primary/Active/Activating -1901> 2014-11-08 18:15:35.500399 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.3fs0(38),4.33s2(44) epoch 5776) v5 -- ?+0 0x213bb660 con 0x1c5c0580 -1900> 2014-11-08 18:15:35.500428 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(2 pgs e5776:4.31s0(1),4.36s1(40)) v4 -- ?+0 0x213bad00 con 0x1c5c2680 -1899> 2014-11-08 18:15:35.500439 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_info(1 pgs e5776:4.36s1(40)) v4 -- ?+0 0x25f930c0 con 0x1e50aec0 -1898> 2014-11-08 18:15:35.500532 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_info(1 pgs e5776:4.36s1(40)) v4 -- ?+0 0x22d52b20 con 0x1c7911e0 -1897> 2014-11-08 18:15:35.514723 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.93s0( v 5734'8111 (3458'5082,5734'8111] local-les=5776 n=7627 ec=195 les/c 5752/5753 5774/5774/5730) [9,2,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5721'8087 lcod 5734'8110 mlcod 0'0 undersized+degraded] enter Started/Primary/Active/Activating -1896> 2014-11-08 18:15:35.515053 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_notify(4.9cs2(22),4.94s1(11) epoch 5776) v5 -- ?+0 0x234e0000 con 0x1c5c0580 -1895> 2014-11-08 18:15:35.515072 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_notify(4.a9s0(45) epoch 5776) v5 -- ?+0 0x234e6180 con 0x1e50aec0 -1894> 2014-11-08 18:15:35.515083 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6811/4928 -- pg_notify(4.8fs2(21) epoch 5776) v5 -- ?+0 0x234e0f00 con 0x42c86e0 -1893> 2014-11-08 18:15:35.515093 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.18:6802/1404 -- pg_notify(4.91s1(30) epoch 5776) v5 -- ?+0 0x4537260 con 0x1c7911e0 -1892> 2014-11-08 18:15:35.515107 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_info(4 pgs e5776:4.a0s0(1),4.98s0(1),4.9ds1(2),4.93s0(1)) v4 -- ?+0 0x4530000 con 0x1c5c0580 -1891> 2014-11-08 18:15:35.515111 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.9 192.168.1.18:6811/4928 0 ==== pg_notify(4.8fs2(21) epoch 5776) v5 ==== 0+0+0 (0 0 0) 0x234e0f00 con 0x42c86e0 -1890> 2014-11-08 18:15:35.515125 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 2014-11-08 18:15:35.515088, event: header_read, op: pg_notify(4.8fs2(21) epoch 5776) -1889> 2014-11-08 18:15:35.515132 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 0.000000, event: throttled, op: pg_notify(4.8fs2(21) epoch 5776) -1888> 2014-11-08 18:15:35.515134 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 0.000000, event: all_read, op: pg_notify(4.8fs2(21) epoch 5776) -1887> 2014-11-08 18:15:35.515136 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 2014-11-08 18:15:35.515120, event: dispatched, op: pg_notify(4.8fs2(21) epoch 5776) -1886> 2014-11-08 18:15:35.515139 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 2014-11-08 18:15:35.515139, event: waiting_for_osdmap, op: pg_notify(4.8fs2(21) epoch 5776) -1885> 2014-11-08 18:15:35.515145 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 2014-11-08 18:15:35.515145, event: started, op: pg_notify(4.8fs2(21) epoch 5776) -1884> 2014-11-08 18:15:35.515165 7fc5b9b29700 5 -- op tracker -- seq: 47280, time: 2014-11-08 18:15:35.515165, event: done, op: pg_notify(4.8fs2(21) epoch 5776) -1883> 2014-11-08 18:15:35.526859 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 341 ==== osd_map(5776..5776 src has 4761..5776) v3 ==== 238+0+0 (2420843160 0 0) 0x2302de80 con 0x1c7911e0 -1882> 2014-11-08 18:15:35.526888 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5776,5776], i have 5776, src has [4761,5776] -1881> 2014-11-08 18:15:35.526895 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 342 ==== pg_notify(4.15s1(28) epoch 5776) v5 ==== 2427+0+0 (4156952743 0 0) 0x4537260 con 0x1c7911e0 -1880> 2014-11-08 18:15:35.526903 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526771, event: header_read, op: pg_notify(4.15s1(28) epoch 5776) -1879> 2014-11-08 18:15:35.526909 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526772, event: throttled, op: pg_notify(4.15s1(28) epoch 5776) -1878> 2014-11-08 18:15:35.526912 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526848, event: all_read, op: pg_notify(4.15s1(28) epoch 5776) -1877> 2014-11-08 18:15:35.526915 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526900, event: dispatched, op: pg_notify(4.15s1(28) epoch 5776) -1876> 2014-11-08 18:15:35.526918 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526917, event: waiting_for_osdmap, op: pg_notify(4.15s1(28) epoch 5776) -1875> 2014-11-08 18:15:35.526924 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526924, event: started, op: pg_notify(4.15s1(28) epoch 5776) -1874> 2014-11-08 18:15:35.526946 7fc5b9b29700 5 -- op tracker -- seq: 47281, time: 2014-11-08 18:15:35.526946, event: done, op: pg_notify(4.15s1(28) epoch 5776) -1873> 2014-11-08 18:15:35.535961 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.10 192.168.1.18:6802/1404 343 ==== pg_notify(4.36s2(40) epoch 5776) v5 ==== 3135+0+0 (1187552520 0 0) 0x22d52b20 con 0x1c7911e0 -1872> 2014-11-08 18:15:35.535988 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.535827, event: header_read, op: pg_notify(4.36s2(40) epoch 5776) -1871> 2014-11-08 18:15:35.535995 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.535830, event: throttled, op: pg_notify(4.36s2(40) epoch 5776) -1870> 2014-11-08 18:15:35.535998 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.535939, event: all_read, op: pg_notify(4.36s2(40) epoch 5776) -1869> 2014-11-08 18:15:35.536000 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.535982, event: dispatched, op: pg_notify(4.36s2(40) epoch 5776) -1868> 2014-11-08 18:15:35.536004 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.536004, event: waiting_for_osdmap, op: pg_notify(4.36s2(40) epoch 5776) -1867> 2014-11-08 18:15:35.536012 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.536012, event: started, op: pg_notify(4.36s2(40) epoch 5776) -1866> 2014-11-08 18:15:35.536036 7fc5b9b29700 5 -- op tracker -- seq: 47282, time: 2014-11-08 18:15:35.536036, event: done, op: pg_notify(4.36s2(40) epoch 5776) -1865> 2014-11-08 18:15:35.537552 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.24s1) v4 -- ?+0 0x1e896720 con 0x1c5c2680 -1864> 2014-11-08 18:15:35.537579 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.99s1) v4 -- ?+0 0x25f97260 con 0x1c5c2680 -1863> 2014-11-08 18:15:35.537592 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.5as2) v4 -- ?+0 0x25f90000 con 0x1c5c2680 -1862> 2014-11-08 18:15:35.537604 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.57s1) v4 -- ?+0 0x25f97da0 con 0x1c5c2680 -1861> 2014-11-08 18:15:35.537616 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.45s2) v4 -- ?+0 0x2290bc00 con 0x1c5c2680 -1860> 2014-11-08 18:15:35.537628 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.9fs2) v4 -- ?+0 0x23d712c0 con 0x1c5c2680 -1859> 2014-11-08 18:15:35.537640 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.8ds1) v4 -- ?+0 0x1d4f1680 con 0x1c5c2680 -1858> 2014-11-08 18:15:35.539448 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.ds1) v4 -- ?+0 0x2285fbc0 con 0x1c5c2680 -1857> 2014-11-08 18:15:35.539474 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.7s2) v4 -- ?+0 0x234e6720 con 0x1c5c2680 -1856> 2014-11-08 18:15:35.539492 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.6ds2) v4 -- ?+0 0x1d069fe0 con 0x1c5c2680 -1855> 2014-11-08 18:15:35.539505 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.67s1) v4 -- ?+0 0x1d06bfc0 con 0x1c5c2680 -1854> 2014-11-08 18:15:35.539517 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.67s2) v4 -- ?+0 0x1d06a1c0 con 0x1c5c2680 -1853> 2014-11-08 18:15:35.539530 7fc5c0e2a700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5775:4.59s2) v4 -- ?+0 0x1d06eae0 con 0x1c5c2680 -1852> 2014-11-08 18:15:35.552382 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2303 ==== osd_map(5775..5775 src has 4761..5775) v3 ==== 298+0+0 (4097883411 0 0) 0x22acc380 con 0x1e50aec0 -1851> 2014-11-08 18:15:35.552429 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5775,5775], i have 5776, src has [4761,5775] -1850> 2014-11-08 18:15:35.609712 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2304 ==== pg_trim(4.83 to 5734'8135 e5774) v1 ==== 34+0+0 (1155027519 0 0) 0x234e6180 con 0x1e50aec0 -1849> 2014-11-08 18:15:35.609746 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609604, event: header_read, op: pg_trim(4.83 to 5734'8135 e5774) -1848> 2014-11-08 18:15:35.609755 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609607, event: throttled, op: pg_trim(4.83 to 5734'8135 e5774) -1847> 2014-11-08 18:15:35.609758 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609679, event: all_read, op: pg_trim(4.83 to 5734'8135 e5774) -1846> 2014-11-08 18:15:35.609760 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609741, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5774) -1845> 2014-11-08 18:15:35.609764 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609764, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5774) -1844> 2014-11-08 18:15:35.609770 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609770, event: started, op: pg_trim(4.83 to 5734'8135 e5774) -1843> 2014-11-08 18:15:35.609776 7fc5b9b29700 5 -- op tracker -- seq: 47283, time: 2014-11-08 18:15:35.609776, event: done, op: pg_trim(4.83 to 5734'8135 e5774) -1842> 2014-11-08 18:15:35.609788 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2305 ==== pg_trim(4.83 to 5734'8135 e5774) v1 ==== 34+0+0 (1155027519 0 0) 0x24958b40 con 0x1e50aec0 -1841> 2014-11-08 18:15:35.609795 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609707, event: header_read, op: pg_trim(4.83 to 5734'8135 e5774) -1840> 2014-11-08 18:15:35.609798 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609707, event: throttled, op: pg_trim(4.83 to 5734'8135 e5774) -1839> 2014-11-08 18:15:35.609801 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609742, event: all_read, op: pg_trim(4.83 to 5734'8135 e5774) -1838> 2014-11-08 18:15:35.609803 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609794, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5774) -1837> 2014-11-08 18:15:35.609806 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609806, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5774) -1836> 2014-11-08 18:15:35.609809 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609809, event: started, op: pg_trim(4.83 to 5734'8135 e5774) -1835> 2014-11-08 18:15:35.609812 7fc5b9b29700 5 -- op tracker -- seq: 47284, time: 2014-11-08 18:15:35.609812, event: done, op: pg_trim(4.83 to 5734'8135 e5774) -1834> 2014-11-08 18:15:35.610120 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2306 ==== pg_trim(4.83 to 5734'8135 e5774) v1 ==== 34+0+0 (1155027519 0 0) 0x249592c0 con 0x1e50aec0 -1833> 2014-11-08 18:15:35.610130 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610085, event: header_read, op: pg_trim(4.83 to 5734'8135 e5774) -1832> 2014-11-08 18:15:35.610133 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610086, event: throttled, op: pg_trim(4.83 to 5734'8135 e5774) -1831> 2014-11-08 18:15:35.610136 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610109, event: all_read, op: pg_trim(4.83 to 5734'8135 e5774) -1830> 2014-11-08 18:15:35.610138 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610129, event: dispatched, op: pg_trim(4.83 to 5734'8135 e5774) -1829> 2014-11-08 18:15:35.610141 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610141, event: waiting_for_osdmap, op: pg_trim(4.83 to 5734'8135 e5774) -1828> 2014-11-08 18:15:35.610145 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610145, event: started, op: pg_trim(4.83 to 5734'8135 e5774) -1827> 2014-11-08 18:15:35.610148 7fc5b9b29700 5 -- op tracker -- seq: 47285, time: 2014-11-08 18:15:35.610148, event: done, op: pg_trim(4.83 to 5734'8135 e5774) -1826> 2014-11-08 18:15:35.926170 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2307 ==== MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) v1 ==== 4195327+0+0 (903307247 0 0) 0x224bee00 con 0x1e50aec0 -1825> 2014-11-08 18:15:35.926217 7fc5a56ef700 5 -- op tracker -- seq: 47286, time: 2014-11-08 18:15:35.838381, event: header_read, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) -1824> 2014-11-08 18:15:35.926232 7fc5a56ef700 5 -- op tracker -- seq: 47286, time: 2014-11-08 18:15:35.838391, event: throttled, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) -1823> 2014-11-08 18:15:35.926241 7fc5a56ef700 5 -- op tracker -- seq: 47286, time: 2014-11-08 18:15:35.926158, event: all_read, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) -1822> 2014-11-08 18:15:35.926250 7fc5a56ef700 5 -- op tracker -- seq: 47286, time: 0.000000, event: dispatched, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) -1821> 2014-11-08 18:15:35.926295 7fc5afb15700 5 -- op tracker -- seq: 47286, time: 2014-11-08 18:15:35.926295, event: reached_pg, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) -1820> 2014-11-08 18:15:35.929795 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.2 192.168.1.15:0/1606 3689 ==== osd_ping(ping e5774 stamp 2014-11-08 18:15:35.926068) v2 ==== 47+0+0 (4260665659 0 0) 0x1caa8960 con 0x1cce2ec0 -1819> 2014-11-08 18:15:35.929849 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5776 stamp 2014-11-08 18:15:35.926068) v2 -- ?+0 0x257c1e00 con 0x1cce2ec0 -1818> 2014-11-08 18:15:35.929881 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.2 192.168.1.15:0/1606 3689 ==== osd_ping(ping e5774 stamp 2014-11-08 18:15:35.926068) v2 ==== 47+0+0 (4260665659 0 0) 0x23682d00 con 0x1cce3440 -1817> 2014-11-08 18:15:35.929891 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.15:0/1606 -- osd_ping(ping_reply e5776 stamp 2014-11-08 18:15:35.926068) v2 -- ?+0 0x22d50f00 con 0x1cce3440 -1816> 2014-11-08 18:15:35.936437 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- MOSDPGPush(4.43s2 5776 [PushOp(5981f343/1000002ab09.00000020/head//4, version: 3467'11315, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5981f343/1000002ab09.00000020/head//4@3467'11315, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(d7c1f343/1000002aa96.00000035/head//4, version: 3458'11234, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(d7c1f343/1000002aa96.00000035/head//4@3458'11234, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 -- ?+0 0x1d3af400 con 0x1c5c0580 -1815> 2014-11-08 18:15:35.936513 7fc5afb15700 5 -- op tracker -- seq: 47286, time: 2014-11-08 18:15:35.936512, event: done, op: MOSDECSubOpReadReply(4.43s0 5774 ECSubReadReply(tid=4428, attrs_read=2)) -1814> 2014-11-08 18:15:35.990355 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2308 ==== pg_notify(4.21s0(100) epoch 5775) v5 ==== 6675+0+0 (3260052623 0 0) 0x2495ecc0 con 0x1e50aec0 -1813> 2014-11-08 18:15:35.990377 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990112, event: header_read, op: pg_notify(4.21s0(100) epoch 5775) -1812> 2014-11-08 18:15:35.990384 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990115, event: throttled, op: pg_notify(4.21s0(100) epoch 5775) -1811> 2014-11-08 18:15:35.990388 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990335, event: all_read, op: pg_notify(4.21s0(100) epoch 5775) -1810> 2014-11-08 18:15:35.990392 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990373, event: dispatched, op: pg_notify(4.21s0(100) epoch 5775) -1809> 2014-11-08 18:15:35.990397 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990397, event: waiting_for_osdmap, op: pg_notify(4.21s0(100) epoch 5775) -1808> 2014-11-08 18:15:35.990404 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990404, event: started, op: pg_notify(4.21s0(100) epoch 5775) -1807> 2014-11-08 18:15:35.990424 7fc5b9b29700 5 -- op tracker -- seq: 47287, time: 2014-11-08 18:15:35.990424, event: done, op: pg_notify(4.21s0(100) epoch 5775) -1806> 2014-11-08 18:15:35.994215 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2309 ==== pg_notify(4.bfs2(127) epoch 5775) v5 ==== 8248+0+0 (3171087091 0 0) 0x2495b2a0 con 0x1e50aec0 -1805> 2014-11-08 18:15:35.994238 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994022, event: header_read, op: pg_notify(4.bfs2(127) epoch 5775) -1804> 2014-11-08 18:15:35.994246 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994025, event: throttled, op: pg_notify(4.bfs2(127) epoch 5775) -1803> 2014-11-08 18:15:35.994250 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994200, event: all_read, op: pg_notify(4.bfs2(127) epoch 5775) -1802> 2014-11-08 18:15:35.994254 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994235, event: dispatched, op: pg_notify(4.bfs2(127) epoch 5775) -1801> 2014-11-08 18:15:35.994259 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994259, event: waiting_for_osdmap, op: pg_notify(4.bfs2(127) epoch 5775) -1800> 2014-11-08 18:15:35.994268 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994268, event: started, op: pg_notify(4.bfs2(127) epoch 5775) -1799> 2014-11-08 18:15:35.994289 7fc5b9b29700 5 -- op tracker -- seq: 47288, time: 2014-11-08 18:15:35.994289, event: done, op: pg_notify(4.bfs2(127) epoch 5775) -1798> 2014-11-08 18:15:35.994397 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2310 ==== pg_notify(4.38s2(53) epoch 5775) v5 ==== 3902+0+0 (3076604678 0 0) 0x24959860 con 0x1e50aec0 -1797> 2014-11-08 18:15:35.994408 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994215, event: header_read, op: pg_notify(4.38s2(53) epoch 5775) -1796> 2014-11-08 18:15:35.994413 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994216, event: throttled, op: pg_notify(4.38s2(53) epoch 5775) -1795> 2014-11-08 18:15:35.994417 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994333, event: all_read, op: pg_notify(4.38s2(53) epoch 5775) -1794> 2014-11-08 18:15:35.994421 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994406, event: dispatched, op: pg_notify(4.38s2(53) epoch 5775) -1793> 2014-11-08 18:15:35.994425 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994425, event: waiting_for_osdmap, op: pg_notify(4.38s2(53) epoch 5775) -1792> 2014-11-08 18:15:35.994431 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994430, event: started, op: pg_notify(4.38s2(53) epoch 5775) -1791> 2014-11-08 18:15:35.994463 7fc5b9b29700 5 -- op tracker -- seq: 47289, time: 2014-11-08 18:15:35.994463, event: done, op: pg_notify(4.38s2(53) epoch 5775) -1790> 2014-11-08 18:15:35.994704 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2311 ==== pg_notify(4.36s2(126) epoch 5775) v5 ==== 8189+0+0 (2598356198 0 0) 0x2495e180 con 0x1e50aec0 -1789> 2014-11-08 18:15:35.994715 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994341, event: header_read, op: pg_notify(4.36s2(126) epoch 5775) -1788> 2014-11-08 18:15:35.994719 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994342, event: throttled, op: pg_notify(4.36s2(126) epoch 5775) -1787> 2014-11-08 18:15:35.994721 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994520, event: all_read, op: pg_notify(4.36s2(126) epoch 5775) -1786> 2014-11-08 18:15:35.994724 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994714, event: dispatched, op: pg_notify(4.36s2(126) epoch 5775) -1785> 2014-11-08 18:15:35.994726 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994726, event: waiting_for_osdmap, op: pg_notify(4.36s2(126) epoch 5775) -1784> 2014-11-08 18:15:35.994730 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994730, event: started, op: pg_notify(4.36s2(126) epoch 5775) -1783> 2014-11-08 18:15:35.994743 7fc5b9b29700 5 -- op tracker -- seq: 47290, time: 2014-11-08 18:15:35.994743, event: done, op: pg_notify(4.36s2(126) epoch 5775) -1782> 2014-11-08 18:15:35.994794 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2312 ==== osd_map(5776..5776 src has 4761..5776) v3 ==== 238+0+0 (2420843160 0 0) 0x224a9f80 con 0x1e50aec0 -1781> 2014-11-08 18:15:35.994809 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5776,5776], i have 5776, src has [4761,5776] -1780> 2014-11-08 18:15:35.994815 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2313 ==== pg_notify(4.21s0(100) epoch 5776) v5 ==== 6675+0+0 (440936794 0 0) 0x2495f9e0 con 0x1e50aec0 -1779> 2014-11-08 18:15:35.994821 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994557, event: header_read, op: pg_notify(4.21s0(100) epoch 5776) -1778> 2014-11-08 18:15:35.994824 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994558, event: throttled, op: pg_notify(4.21s0(100) epoch 5776) -1777> 2014-11-08 18:15:35.994826 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994649, event: all_read, op: pg_notify(4.21s0(100) epoch 5776) -1776> 2014-11-08 18:15:35.994829 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994820, event: dispatched, op: pg_notify(4.21s0(100) epoch 5776) -1775> 2014-11-08 18:15:35.994831 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994831, event: waiting_for_osdmap, op: pg_notify(4.21s0(100) epoch 5776) -1774> 2014-11-08 18:15:35.994835 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994834, event: started, op: pg_notify(4.21s0(100) epoch 5776) -1773> 2014-11-08 18:15:35.994848 7fc5b9b29700 5 -- op tracker -- seq: 47291, time: 2014-11-08 18:15:35.994848, event: done, op: pg_notify(4.21s0(100) epoch 5776) -1772> 2014-11-08 18:15:35.996989 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2314 ==== pg_notify(4.bfs2(127) epoch 5776) v5 ==== 8248+0+0 (785102492 0 0) 0x24959c20 con 0x1e50aec0 -1771> 2014-11-08 18:15:35.997003 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.996851, event: header_read, op: pg_notify(4.bfs2(127) epoch 5776) -1770> 2014-11-08 18:15:35.997007 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.996853, event: throttled, op: pg_notify(4.bfs2(127) epoch 5776) -1769> 2014-11-08 18:15:35.997009 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.996975, event: all_read, op: pg_notify(4.bfs2(127) epoch 5776) -1768> 2014-11-08 18:15:35.997012 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.997001, event: dispatched, op: pg_notify(4.bfs2(127) epoch 5776) -1767> 2014-11-08 18:15:35.997015 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.997014, event: waiting_for_osdmap, op: pg_notify(4.bfs2(127) epoch 5776) -1766> 2014-11-08 18:15:35.997019 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.997019, event: started, op: pg_notify(4.bfs2(127) epoch 5776) -1765> 2014-11-08 18:15:35.997032 7fc5b9b29700 5 -- op tracker -- seq: 47292, time: 2014-11-08 18:15:35.997032, event: done, op: pg_notify(4.bfs2(127) epoch 5776) -1764> 2014-11-08 18:15:36.000590 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2315 ==== pg_notify(4.38s2(53),4.36s2(126) epoch 5776) v5 ==== 12079+0+0 (721314701 0 0) 0x24959a40 con 0x1e50aec0 -1763> 2014-11-08 18:15:36.000602 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:35.999276, event: header_read, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1762> 2014-11-08 18:15:36.000606 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:35.999277, event: throttled, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1761> 2014-11-08 18:15:36.000609 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:35.999498, event: all_read, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1760> 2014-11-08 18:15:36.000612 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:36.000601, event: dispatched, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1759> 2014-11-08 18:15:36.000615 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:36.000615, event: waiting_for_osdmap, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1758> 2014-11-08 18:15:36.000619 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:36.000619, event: started, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1757> 2014-11-08 18:15:36.000640 7fc5b9b29700 5 -- op tracker -- seq: 47293, time: 2014-11-08 18:15:36.000640, event: done, op: pg_notify(4.38s2(53),4.36s2(126) epoch 5776) -1756> 2014-11-08 18:15:36.370508 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12104 ==== pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) v2 ==== 139+0+0 (53306842 0 0) 0x22acc140 con 0x1c5c2680 -1755> 2014-11-08 18:15:36.370544 7fc5a6cfe700 5 -- op tracker -- seq: 47294, time: 2014-11-08 18:15:36.370414, event: header_read, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1754> 2014-11-08 18:15:36.370556 7fc5a6cfe700 5 -- op tracker -- seq: 47294, time: 2014-11-08 18:15:36.370418, event: throttled, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1753> 2014-11-08 18:15:36.370617 7fc5a6cfe700 5 -- op tracker -- seq: 47294, time: 2014-11-08 18:15:36.370498, event: all_read, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1752> 2014-11-08 18:15:36.370621 7fc5a6cfe700 5 -- op tracker -- seq: 47294, time: 0.000000, event: dispatched, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1751> 2014-11-08 18:15:36.370662 7fc5b0316700 5 -- op tracker -- seq: 47294, time: 2014-11-08 18:15:36.370662, event: reached_pg, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1750> 2014-11-08 18:15:36.370675 7fc5b0316700 5 -- op tracker -- seq: 47294, time: 2014-11-08 18:15:36.370675, event: started, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1749> 2014-11-08 18:15:36.374970 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12105 ==== osd_map(5776..5776 src has 4761..5776) v3 ==== 238+0+0 (2420843160 0 0) 0x228a06c0 con 0x1c5c2680 -1748> 2014-11-08 18:15:36.375001 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5776,5776], i have 5776, src has [4761,5776] -1747> 2014-11-08 18:15:36.375008 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12106 ==== pg_notify(4.es2(34),4.bfs0(41) epoch 5776) v5 ==== 5923+0+0 (2170971093 0 0) 0x1d06eae0 con 0x1c5c2680 -1746> 2014-11-08 18:15:36.375015 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.374823, event: header_read, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1745> 2014-11-08 18:15:36.375021 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.374823, event: throttled, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1744> 2014-11-08 18:15:36.375023 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.374936, event: all_read, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1743> 2014-11-08 18:15:36.375026 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.375013, event: dispatched, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1742> 2014-11-08 18:15:36.375029 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.375029, event: waiting_for_osdmap, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1741> 2014-11-08 18:15:36.375036 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.375036, event: started, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1740> 2014-11-08 18:15:36.375066 7fc5b9b29700 5 -- op tracker -- seq: 47295, time: 2014-11-08 18:15:36.375066, event: done, op: pg_notify(4.es2(34),4.bfs0(41) epoch 5776) -1739> 2014-11-08 18:15:36.375560 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12107 ==== pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) v5 ==== 3170+0+0 (3977802382 0 0) 0x1d06a1c0 con 0x1c5c2680 -1738> 2014-11-08 18:15:36.375572 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.374959, event: header_read, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1737> 2014-11-08 18:15:36.375576 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.374959, event: throttled, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1736> 2014-11-08 18:15:36.375579 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.375548, event: all_read, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1735> 2014-11-08 18:15:36.375582 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.375571, event: dispatched, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1734> 2014-11-08 18:15:36.375585 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.375585, event: waiting_for_osdmap, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1733> 2014-11-08 18:15:36.375589 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.375589, event: started, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1732> 2014-11-08 18:15:36.375612 7fc5b9b29700 5 -- op tracker -- seq: 47296, time: 2014-11-08 18:15:36.375612, event: done, op: pg_notify(4.8fs1(21),4.81s1(7) epoch 5776) -1731> 2014-11-08 18:15:36.375905 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12108 ==== pg_notify(4.75s1(7) epoch 5776) v5 ==== 1168+0+0 (977574526 0 0) 0x1d06bfc0 con 0x1c5c2680 -1730> 2014-11-08 18:15:36.375916 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375852, event: header_read, op: pg_notify(4.75s1(7) epoch 5776) -1729> 2014-11-08 18:15:36.375920 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375853, event: throttled, op: pg_notify(4.75s1(7) epoch 5776) -1728> 2014-11-08 18:15:36.375925 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375890, event: all_read, op: pg_notify(4.75s1(7) epoch 5776) -1727> 2014-11-08 18:15:36.375927 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375915, event: dispatched, op: pg_notify(4.75s1(7) epoch 5776) -1726> 2014-11-08 18:15:36.375932 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375932, event: waiting_for_osdmap, op: pg_notify(4.75s1(7) epoch 5776) -1725> 2014-11-08 18:15:36.375936 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375936, event: started, op: pg_notify(4.75s1(7) epoch 5776) -1724> 2014-11-08 18:15:36.375951 7fc5b9b29700 5 -- op tracker -- seq: 47297, time: 2014-11-08 18:15:36.375951, event: done, op: pg_notify(4.75s1(7) epoch 5776) -1723> 2014-11-08 18:15:36.377140 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12109 ==== pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) v5 ==== 15095+0+0 (2795996131 0 0) 0x1d069fe0 con 0x1c5c2680 -1722> 2014-11-08 18:15:36.377156 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.376623, event: header_read, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1721> 2014-11-08 18:15:36.377163 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.376624, event: throttled, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1720> 2014-11-08 18:15:36.377169 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.377124, event: all_read, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1719> 2014-11-08 18:15:36.377175 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.377154, event: dispatched, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1718> 2014-11-08 18:15:36.377180 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.377180, event: waiting_for_osdmap, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1717> 2014-11-08 18:15:36.377189 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.377189, event: started, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1716> 2014-11-08 18:15:36.377236 7fc5b9b29700 5 -- op tracker -- seq: 47298, time: 2014-11-08 18:15:36.377236, event: done, op: pg_notify(4.13s1(40),4.15s2(83),4.15s0(28),4.9s1(7),4.cs1(34) epoch 5776) -1715> 2014-11-08 18:15:36.379002 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12110 ==== pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) v5 ==== 6312+0+0 (1609585162 0 0) 0x234e6720 con 0x1c5c2680 -1714> 2014-11-08 18:15:36.379017 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.378890, event: header_read, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1713> 2014-11-08 18:15:36.379024 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.378892, event: throttled, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1712> 2014-11-08 18:15:36.379028 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.378988, event: all_read, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1711> 2014-11-08 18:15:36.379033 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.379015, event: dispatched, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1710> 2014-11-08 18:15:36.379043 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.379043, event: waiting_for_osdmap, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1709> 2014-11-08 18:15:36.379051 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.379051, event: started, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1708> 2014-11-08 18:15:36.379117 7fc5b9b29700 5 -- op tracker -- seq: 47299, time: 2014-11-08 18:15:36.379117, event: done, op: pg_notify(4.39s1(7),4.31s2(22),4.36s0(40) epoch 5776) -1707> 2014-11-08 18:15:36.382300 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12111 ==== pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) v5 ==== 7748+0+0 (29733972 0 0) 0x2285fbc0 con 0x1c5c2680 -1706> 2014-11-08 18:15:36.382325 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382122, event: header_read, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1705> 2014-11-08 18:15:36.382333 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382125, event: throttled, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1704> 2014-11-08 18:15:36.382337 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382282, event: all_read, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1703> 2014-11-08 18:15:36.382377 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382321, event: dispatched, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1702> 2014-11-08 18:15:36.382383 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382383, event: waiting_for_osdmap, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1701> 2014-11-08 18:15:36.382391 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382391, event: started, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1700> 2014-11-08 18:15:36.382427 7fc5b9b29700 5 -- op tracker -- seq: 47300, time: 2014-11-08 18:15:36.382427, event: done, op: pg_notify(4.4bs0(44),4.4es2(19),4.40s1(30) epoch 5776) -1699> 2014-11-08 18:15:36.391118 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12112 ==== pg_info(1 pgs e5776:4.99s0) v4 ==== 751+0+0 (2111999063 0 0) 0x1d4f1680 con 0x1c5c2680 -1698> 2014-11-08 18:15:36.391140 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391021, event: header_read, op: pg_info(1 pgs e5776:4.99s0) -1697> 2014-11-08 18:15:36.391146 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391024, event: throttled, op: pg_info(1 pgs e5776:4.99s0) -1696> 2014-11-08 18:15:36.391148 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391100, event: all_read, op: pg_info(1 pgs e5776:4.99s0) -1695> 2014-11-08 18:15:36.391151 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391136, event: dispatched, op: pg_info(1 pgs e5776:4.99s0) -1694> 2014-11-08 18:15:36.391154 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391154, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.99s0) -1693> 2014-11-08 18:15:36.391161 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391161, event: started, op: pg_info(1 pgs e5776:4.99s0) -1692> 2014-11-08 18:15:36.391180 7fc5b9b29700 5 -- op tracker -- seq: 47301, time: 2014-11-08 18:15:36.391180, event: done, op: pg_info(1 pgs e5776:4.99s0) -1691> 2014-11-08 18:15:36.391355 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12113 ==== pg_info(1 pgs e5776:4.5as0) v4 ==== 751+0+0 (4276845896 0 0) 0x23d712c0 con 0x1c5c2680 -1690> 2014-11-08 18:15:36.391363 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391319, event: header_read, op: pg_info(1 pgs e5776:4.5as0) -1689> 2014-11-08 18:15:36.391366 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391319, event: throttled, op: pg_info(1 pgs e5776:4.5as0) -1688> 2014-11-08 18:15:36.391369 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391347, event: all_read, op: pg_info(1 pgs e5776:4.5as0) -1687> 2014-11-08 18:15:36.391371 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391362, event: dispatched, op: pg_info(1 pgs e5776:4.5as0) -1686> 2014-11-08 18:15:36.391374 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391374, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.5as0) -1685> 2014-11-08 18:15:36.391377 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391377, event: started, op: pg_info(1 pgs e5776:4.5as0) -1684> 2014-11-08 18:15:36.391390 7fc5b9b29700 5 -- op tracker -- seq: 47302, time: 2014-11-08 18:15:36.391390, event: done, op: pg_info(1 pgs e5776:4.5as0) -1683> 2014-11-08 18:15:36.391589 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12114 ==== pg_info(1 pgs e5776:4.9fs0) v4 ==== 751+0+0 (2110417983 0 0) 0x2290bc00 con 0x1c5c2680 -1682> 2014-11-08 18:15:36.391597 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391552, event: header_read, op: pg_info(1 pgs e5776:4.9fs0) -1681> 2014-11-08 18:15:36.391600 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391553, event: throttled, op: pg_info(1 pgs e5776:4.9fs0) -1680> 2014-11-08 18:15:36.391603 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391581, event: all_read, op: pg_info(1 pgs e5776:4.9fs0) -1679> 2014-11-08 18:15:36.391605 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391596, event: dispatched, op: pg_info(1 pgs e5776:4.9fs0) -1678> 2014-11-08 18:15:36.391608 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391608, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.9fs0) -1677> 2014-11-08 18:15:36.391611 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391611, event: started, op: pg_info(1 pgs e5776:4.9fs0) -1676> 2014-11-08 18:15:36.391623 7fc5b9b29700 5 -- op tracker -- seq: 47303, time: 2014-11-08 18:15:36.391623, event: done, op: pg_info(1 pgs e5776:4.9fs0) -1675> 2014-11-08 18:15:36.392220 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12115 ==== pg_info(1 pgs e5776:4.8ds0) v4 ==== 751+0+0 (679082937 0 0) 0x25f97da0 con 0x1c5c2680 -1674> 2014-11-08 18:15:36.392231 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392086, event: header_read, op: pg_info(1 pgs e5776:4.8ds0) -1673> 2014-11-08 18:15:36.392235 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392086, event: throttled, op: pg_info(1 pgs e5776:4.8ds0) -1672> 2014-11-08 18:15:36.392237 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392116, event: all_read, op: pg_info(1 pgs e5776:4.8ds0) -1671> 2014-11-08 18:15:36.392240 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392230, event: dispatched, op: pg_info(1 pgs e5776:4.8ds0) -1670> 2014-11-08 18:15:36.392243 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392243, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.8ds0) -1669> 2014-11-08 18:15:36.392246 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392246, event: started, op: pg_info(1 pgs e5776:4.8ds0) -1668> 2014-11-08 18:15:36.392259 7fc5b9b29700 5 -- op tracker -- seq: 47304, time: 2014-11-08 18:15:36.392259, event: done, op: pg_info(1 pgs e5776:4.8ds0) -1667> 2014-11-08 18:15:36.392266 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12116 ==== pg_info(1 pgs e5776:4.ds0) v4 ==== 751+0+0 (1014778210 0 0) 0x25f90000 con 0x1c5c2680 -1666> 2014-11-08 18:15:36.392272 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392124, event: header_read, op: pg_info(1 pgs e5776:4.ds0) -1665> 2014-11-08 18:15:36.392275 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392124, event: throttled, op: pg_info(1 pgs e5776:4.ds0) -1664> 2014-11-08 18:15:36.392278 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392155, event: all_read, op: pg_info(1 pgs e5776:4.ds0) -1663> 2014-11-08 18:15:36.392280 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392271, event: dispatched, op: pg_info(1 pgs e5776:4.ds0) -1662> 2014-11-08 18:15:36.392282 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392282, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.ds0) -1661> 2014-11-08 18:15:36.392286 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392286, event: started, op: pg_info(1 pgs e5776:4.ds0) -1660> 2014-11-08 18:15:36.392298 7fc5b9b29700 5 -- op tracker -- seq: 47305, time: 2014-11-08 18:15:36.392298, event: done, op: pg_info(1 pgs e5776:4.ds0) -1659> 2014-11-08 18:15:36.392304 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12117 ==== pg_info(1 pgs e5776:4.7s1) v4 ==== 751+0+0 (3027125370 0 0) 0x25f97260 con 0x1c5c2680 -1658> 2014-11-08 18:15:36.392310 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392160, event: header_read, op: pg_info(1 pgs e5776:4.7s1) -1657> 2014-11-08 18:15:36.392313 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392161, event: throttled, op: pg_info(1 pgs e5776:4.7s1) -1656> 2014-11-08 18:15:36.392315 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392185, event: all_read, op: pg_info(1 pgs e5776:4.7s1) -1655> 2014-11-08 18:15:36.392318 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392309, event: dispatched, op: pg_info(1 pgs e5776:4.7s1) -1654> 2014-11-08 18:15:36.392320 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392320, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.7s1) -1653> 2014-11-08 18:15:36.392324 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392324, event: started, op: pg_info(1 pgs e5776:4.7s1) -1652> 2014-11-08 18:15:36.392333 7fc5b9b29700 5 -- op tracker -- seq: 47306, time: 2014-11-08 18:15:36.392333, event: done, op: pg_info(1 pgs e5776:4.7s1) -1651> 2014-11-08 18:15:36.392339 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12118 ==== pg_info(1 pgs e5776:4.6ds0) v4 ==== 751+0+0 (2416898538 0 0) 0x1e896720 con 0x1c5c2680 -1650> 2014-11-08 18:15:36.392344 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392190, event: header_read, op: pg_info(1 pgs e5776:4.6ds0) -1649> 2014-11-08 18:15:36.392347 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392190, event: throttled, op: pg_info(1 pgs e5776:4.6ds0) -1648> 2014-11-08 18:15:36.392349 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392214, event: all_read, op: pg_info(1 pgs e5776:4.6ds0) -1647> 2014-11-08 18:15:36.392352 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392343, event: dispatched, op: pg_info(1 pgs e5776:4.6ds0) -1646> 2014-11-08 18:15:36.392354 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392354, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.6ds0) -1645> 2014-11-08 18:15:36.392357 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392357, event: started, op: pg_info(1 pgs e5776:4.6ds0) -1644> 2014-11-08 18:15:36.392366 7fc5b9b29700 5 -- op tracker -- seq: 47307, time: 2014-11-08 18:15:36.392366, event: done, op: pg_info(1 pgs e5776:4.6ds0) -1643> 2014-11-08 18:15:36.392544 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12119 ==== pg_info(1 pgs e5776:4.24s0) v4 ==== 751+0+0 (2404268470 0 0) 0x213bad00 con 0x1c5c2680 -1642> 2014-11-08 18:15:36.392551 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392508, event: header_read, op: pg_info(1 pgs e5776:4.24s0) -1641> 2014-11-08 18:15:36.392555 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392508, event: throttled, op: pg_info(1 pgs e5776:4.24s0) -1640> 2014-11-08 18:15:36.392631 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392536, event: all_read, op: pg_info(1 pgs e5776:4.24s0) -1639> 2014-11-08 18:15:36.392634 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392550, event: dispatched, op: pg_info(1 pgs e5776:4.24s0) -1638> 2014-11-08 18:15:36.392636 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392636, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.24s0) -1637> 2014-11-08 18:15:36.392640 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392640, event: started, op: pg_info(1 pgs e5776:4.24s0) -1636> 2014-11-08 18:15:36.392653 7fc5b9b29700 5 -- op tracker -- seq: 47308, time: 2014-11-08 18:15:36.392653, event: done, op: pg_info(1 pgs e5776:4.24s0) -1635> 2014-11-08 18:15:36.392824 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12120 ==== pg_info(1 pgs e5776:4.57s0) v4 ==== 751+0+0 (545739502 0 0) 0x213beae0 con 0x1c5c2680 -1634> 2014-11-08 18:15:36.392833 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392784, event: header_read, op: pg_info(1 pgs e5776:4.57s0) -1633> 2014-11-08 18:15:36.392837 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392785, event: throttled, op: pg_info(1 pgs e5776:4.57s0) -1632> 2014-11-08 18:15:36.392840 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392815, event: all_read, op: pg_info(1 pgs e5776:4.57s0) -1631> 2014-11-08 18:15:36.392842 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392832, event: dispatched, op: pg_info(1 pgs e5776:4.57s0) -1630> 2014-11-08 18:15:36.392844 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392844, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.57s0) -1629> 2014-11-08 18:15:36.392848 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392848, event: started, op: pg_info(1 pgs e5776:4.57s0) -1628> 2014-11-08 18:15:36.392860 7fc5b9b29700 5 -- op tracker -- seq: 47309, time: 2014-11-08 18:15:36.392860, event: done, op: pg_info(1 pgs e5776:4.57s0) -1627> 2014-11-08 18:15:36.392867 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12121 ==== pg_info(1 pgs e5776:4.67s1) v4 ==== 751+0+0 (2261524258 0 0) 0x1e536ae0 con 0x1c5c2680 -1626> 2014-11-08 18:15:36.392907 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392825, event: header_read, op: pg_info(1 pgs e5776:4.67s1) -1625> 2014-11-08 18:15:36.392912 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392825, event: throttled, op: pg_info(1 pgs e5776:4.67s1) -1624> 2014-11-08 18:15:36.392917 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392850, event: all_read, op: pg_info(1 pgs e5776:4.67s1) -1623> 2014-11-08 18:15:36.392921 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392906, event: dispatched, op: pg_info(1 pgs e5776:4.67s1) -1622> 2014-11-08 18:15:36.392926 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392926, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.67s1) -1621> 2014-11-08 18:15:36.392931 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392931, event: started, op: pg_info(1 pgs e5776:4.67s1) -1620> 2014-11-08 18:15:36.392945 7fc5b9b29700 5 -- op tracker -- seq: 47310, time: 2014-11-08 18:15:36.392945, event: done, op: pg_info(1 pgs e5776:4.67s1) -1619> 2014-11-08 18:15:36.395863 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12122 ==== pg_info(1 pgs e5776:4.45s1) v4 ==== 751+0+0 (3127721624 0 0) 0x21278780 con 0x1c5c2680 -1618> 2014-11-08 18:15:36.395875 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395810, event: header_read, op: pg_info(1 pgs e5776:4.45s1) -1617> 2014-11-08 18:15:36.395882 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395812, event: throttled, op: pg_info(1 pgs e5776:4.45s1) -1616> 2014-11-08 18:15:36.395886 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395852, event: all_read, op: pg_info(1 pgs e5776:4.45s1) -1615> 2014-11-08 18:15:36.395889 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395874, event: dispatched, op: pg_info(1 pgs e5776:4.45s1) -1614> 2014-11-08 18:15:36.395894 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395894, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.45s1) -1613> 2014-11-08 18:15:36.395898 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395898, event: started, op: pg_info(1 pgs e5776:4.45s1) -1612> 2014-11-08 18:15:36.395913 7fc5b9b29700 5 -- op tracker -- seq: 47311, time: 2014-11-08 18:15:36.395913, event: done, op: pg_info(1 pgs e5776:4.45s1) -1611> 2014-11-08 18:15:36.396132 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12123 ==== pg_info(1 pgs e5776:4.67s1) v4 ==== 751+0+0 (2510089430 0 0) 0x234e1fe0 con 0x1c5c2680 -1610> 2014-11-08 18:15:36.396143 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396087, event: header_read, op: pg_info(1 pgs e5776:4.67s1) -1609> 2014-11-08 18:15:36.396149 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396087, event: throttled, op: pg_info(1 pgs e5776:4.67s1) -1608> 2014-11-08 18:15:36.396151 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396116, event: all_read, op: pg_info(1 pgs e5776:4.67s1) -1607> 2014-11-08 18:15:36.396155 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396141, event: dispatched, op: pg_info(1 pgs e5776:4.67s1) -1606> 2014-11-08 18:15:36.396159 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396159, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.67s1) -1605> 2014-11-08 18:15:36.396164 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396164, event: started, op: pg_info(1 pgs e5776:4.67s1) -1604> 2014-11-08 18:15:36.396179 7fc5b9b29700 5 -- op tracker -- seq: 47312, time: 2014-11-08 18:15:36.396179, event: done, op: pg_info(1 pgs e5776:4.67s1) -1603> 2014-11-08 18:15:36.396258 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12124 ==== pg_info(1 pgs e5776:4.59s1) v4 ==== 751+0+0 (1844246085 0 0) 0x234e32a0 con 0x1c5c2680 -1602> 2014-11-08 18:15:36.396268 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396128, event: header_read, op: pg_info(1 pgs e5776:4.59s1) -1601> 2014-11-08 18:15:36.396271 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396128, event: throttled, op: pg_info(1 pgs e5776:4.59s1) -1600> 2014-11-08 18:15:36.396274 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396249, event: all_read, op: pg_info(1 pgs e5776:4.59s1) -1599> 2014-11-08 18:15:36.396276 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396266, event: dispatched, op: pg_info(1 pgs e5776:4.59s1) -1598> 2014-11-08 18:15:36.396279 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396279, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.59s1) -1597> 2014-11-08 18:15:36.396283 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396283, event: started, op: pg_info(1 pgs e5776:4.59s1) -1596> 2014-11-08 18:15:36.396295 7fc5b9b29700 5 -- op tracker -- seq: 47313, time: 2014-11-08 18:15:36.396295, event: done, op: pg_info(1 pgs e5776:4.59s1) -1595> 2014-11-08 18:15:36.400912 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12125 ==== pg_info(1 pgs e5776:4.24s0) v4 ==== 751+0+0 (2509553927 0 0) 0x21278f00 con 0x1c5c2680 -1594> 2014-11-08 18:15:36.400929 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400842, event: header_read, op: pg_info(1 pgs e5776:4.24s0) -1593> 2014-11-08 18:15:36.400936 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400845, event: throttled, op: pg_info(1 pgs e5776:4.24s0) -1592> 2014-11-08 18:15:36.400939 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400898, event: all_read, op: pg_info(1 pgs e5776:4.24s0) -1591> 2014-11-08 18:15:36.400943 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400927, event: dispatched, op: pg_info(1 pgs e5776:4.24s0) -1590> 2014-11-08 18:15:36.400947 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400947, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.24s0) -1589> 2014-11-08 18:15:36.400954 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400954, event: started, op: pg_info(1 pgs e5776:4.24s0) -1588> 2014-11-08 18:15:36.400971 7fc5b9b29700 5 -- op tracker -- seq: 47314, time: 2014-11-08 18:15:36.400971, event: done, op: pg_info(1 pgs e5776:4.24s0) -1587> 2014-11-08 18:15:36.401400 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12126 ==== pg_info(1 pgs e5776:4.99s0) v4 ==== 751+0+0 (2112107438 0 0) 0x25f92ee0 con 0x1c5c2680 -1586> 2014-11-08 18:15:36.401410 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401340, event: header_read, op: pg_info(1 pgs e5776:4.99s0) -1585> 2014-11-08 18:15:36.401415 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401340, event: throttled, op: pg_info(1 pgs e5776:4.99s0) -1584> 2014-11-08 18:15:36.401419 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401389, event: all_read, op: pg_info(1 pgs e5776:4.99s0) -1583> 2014-11-08 18:15:36.401423 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401409, event: dispatched, op: pg_info(1 pgs e5776:4.99s0) -1582> 2014-11-08 18:15:36.401427 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401427, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.99s0) -1581> 2014-11-08 18:15:36.401432 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401432, event: started, op: pg_info(1 pgs e5776:4.99s0) -1580> 2014-11-08 18:15:36.401447 7fc5b9b29700 5 -- op tracker -- seq: 47315, time: 2014-11-08 18:15:36.401447, event: done, op: pg_info(1 pgs e5776:4.99s0) -1579> 2014-11-08 18:15:36.401641 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12127 ==== pg_info(1 pgs e5776:4.57s0) v4 ==== 751+0+0 (1024784605 0 0) 0x25f93660 con 0x1c5c2680 -1578> 2014-11-08 18:15:36.401651 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401603, event: header_read, op: pg_info(1 pgs e5776:4.57s0) -1577> 2014-11-08 18:15:36.401657 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401603, event: throttled, op: pg_info(1 pgs e5776:4.57s0) -1576> 2014-11-08 18:15:36.401662 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401632, event: all_read, op: pg_info(1 pgs e5776:4.57s0) -1575> 2014-11-08 18:15:36.401665 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401650, event: dispatched, op: pg_info(1 pgs e5776:4.57s0) -1574> 2014-11-08 18:15:36.401670 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401670, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.57s0) -1573> 2014-11-08 18:15:36.401674 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401674, event: started, op: pg_info(1 pgs e5776:4.57s0) -1572> 2014-11-08 18:15:36.401694 7fc5b9b29700 5 -- op tracker -- seq: 47316, time: 2014-11-08 18:15:36.401694, event: done, op: pg_info(1 pgs e5776:4.57s0) -1571> 2014-11-08 18:15:36.402184 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12128 ==== pg_info(1 pgs e5776:4.45s1) v4 ==== 751+0+0 (2426111068 0 0) 0x25f932a0 con 0x1c5c2680 -1570> 2014-11-08 18:15:36.402194 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402141, event: header_read, op: pg_info(1 pgs e5776:4.45s1) -1569> 2014-11-08 18:15:36.402199 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402142, event: throttled, op: pg_info(1 pgs e5776:4.45s1) -1568> 2014-11-08 18:15:36.402203 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402173, event: all_read, op: pg_info(1 pgs e5776:4.45s1) -1567> 2014-11-08 18:15:36.402208 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402193, event: dispatched, op: pg_info(1 pgs e5776:4.45s1) -1566> 2014-11-08 18:15:36.402212 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402212, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.45s1) -1565> 2014-11-08 18:15:36.402217 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402217, event: started, op: pg_info(1 pgs e5776:4.45s1) -1564> 2014-11-08 18:15:36.402232 7fc5b9b29700 5 -- op tracker -- seq: 47317, time: 2014-11-08 18:15:36.402232, event: done, op: pg_info(1 pgs e5776:4.45s1) -1563> 2014-11-08 18:15:36.402393 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12129 ==== pg_info(1 pgs e5776:4.5as0) v4 ==== 751+0+0 (711271197 0 0) 0x25f95640 con 0x1c5c2680 -1562> 2014-11-08 18:15:36.402403 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402349, event: header_read, op: pg_info(1 pgs e5776:4.5as0) -1561> 2014-11-08 18:15:36.402407 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402350, event: throttled, op: pg_info(1 pgs e5776:4.5as0) -1560> 2014-11-08 18:15:36.402412 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402380, event: all_read, op: pg_info(1 pgs e5776:4.5as0) -1559> 2014-11-08 18:15:36.402416 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402401, event: dispatched, op: pg_info(1 pgs e5776:4.5as0) -1558> 2014-11-08 18:15:36.402420 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402420, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.5as0) -1557> 2014-11-08 18:15:36.402425 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402425, event: started, op: pg_info(1 pgs e5776:4.5as0) -1556> 2014-11-08 18:15:36.402443 7fc5b9b29700 5 -- op tracker -- seq: 47318, time: 2014-11-08 18:15:36.402443, event: done, op: pg_info(1 pgs e5776:4.5as0) -1555> 2014-11-08 18:15:36.402638 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12130 ==== pg_info(1 pgs e5776:4.9fs0) v4 ==== 751+0+0 (671956457 0 0) 0x234e05a0 con 0x1c5c2680 -1554> 2014-11-08 18:15:36.402649 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402596, event: header_read, op: pg_info(1 pgs e5776:4.9fs0) -1553> 2014-11-08 18:15:36.402652 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402597, event: throttled, op: pg_info(1 pgs e5776:4.9fs0) -1552> 2014-11-08 18:15:36.402657 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402626, event: all_read, op: pg_info(1 pgs e5776:4.9fs0) -1551> 2014-11-08 18:15:36.402660 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402647, event: dispatched, op: pg_info(1 pgs e5776:4.9fs0) -1550> 2014-11-08 18:15:36.402664 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402664, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.9fs0) -1549> 2014-11-08 18:15:36.402668 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402668, event: started, op: pg_info(1 pgs e5776:4.9fs0) -1548> 2014-11-08 18:15:36.402682 7fc5b9b29700 5 -- op tracker -- seq: 47319, time: 2014-11-08 18:15:36.402682, event: done, op: pg_info(1 pgs e5776:4.9fs0) -1547> 2014-11-08 18:15:36.402882 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12131 ==== pg_info(1 pgs e5776:4.8ds0) v4 ==== 751+0+0 (3950820787 0 0) 0x234e6ae0 con 0x1c5c2680 -1546> 2014-11-08 18:15:36.402892 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402840, event: header_read, op: pg_info(1 pgs e5776:4.8ds0) -1545> 2014-11-08 18:15:36.402896 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402840, event: throttled, op: pg_info(1 pgs e5776:4.8ds0) -1544> 2014-11-08 18:15:36.402899 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402869, event: all_read, op: pg_info(1 pgs e5776:4.8ds0) -1543> 2014-11-08 18:15:36.402903 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402891, event: dispatched, op: pg_info(1 pgs e5776:4.8ds0) -1542> 2014-11-08 18:15:36.402906 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402906, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.8ds0) -1541> 2014-11-08 18:15:36.402912 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402911, event: started, op: pg_info(1 pgs e5776:4.8ds0) -1540> 2014-11-08 18:15:36.402925 7fc5b9b29700 5 -- op tracker -- seq: 47320, time: 2014-11-08 18:15:36.402925, event: done, op: pg_info(1 pgs e5776:4.8ds0) -1539> 2014-11-08 18:15:36.403163 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12132 ==== pg_info(1 pgs e5776:4.ds0) v4 ==== 751+0+0 (984116100 0 0) 0x234e5fa0 con 0x1c5c2680 -1538> 2014-11-08 18:15:36.403182 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403101, event: header_read, op: pg_info(1 pgs e5776:4.ds0) -1537> 2014-11-08 18:15:36.403187 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403102, event: throttled, op: pg_info(1 pgs e5776:4.ds0) -1536> 2014-11-08 18:15:36.403192 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403147, event: all_read, op: pg_info(1 pgs e5776:4.ds0) -1535> 2014-11-08 18:15:36.403196 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403181, event: dispatched, op: pg_info(1 pgs e5776:4.ds0) -1534> 2014-11-08 18:15:36.403200 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403200, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.ds0) -1533> 2014-11-08 18:15:36.403206 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403206, event: started, op: pg_info(1 pgs e5776:4.ds0) -1532> 2014-11-08 18:15:36.403220 7fc5b9b29700 5 -- op tracker -- seq: 47321, time: 2014-11-08 18:15:36.403220, event: done, op: pg_info(1 pgs e5776:4.ds0) -1531> 2014-11-08 18:15:36.403402 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12133 ==== pg_info(1 pgs e5776:4.7s1) v4 ==== 751+0+0 (1068126895 0 0) 0x225830c0 con 0x1c5c2680 -1530> 2014-11-08 18:15:36.403412 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403361, event: header_read, op: pg_info(1 pgs e5776:4.7s1) -1529> 2014-11-08 18:15:36.403417 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403361, event: throttled, op: pg_info(1 pgs e5776:4.7s1) -1528> 2014-11-08 18:15:36.403421 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403392, event: all_read, op: pg_info(1 pgs e5776:4.7s1) -1527> 2014-11-08 18:15:36.403424 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403411, event: dispatched, op: pg_info(1 pgs e5776:4.7s1) -1526> 2014-11-08 18:15:36.403427 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403427, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.7s1) -1525> 2014-11-08 18:15:36.403430 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403430, event: started, op: pg_info(1 pgs e5776:4.7s1) -1524> 2014-11-08 18:15:36.403442 7fc5b9b29700 5 -- op tracker -- seq: 47322, time: 2014-11-08 18:15:36.403442, event: done, op: pg_info(1 pgs e5776:4.7s1) -1523> 2014-11-08 18:15:36.404909 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12134 ==== pg_info(1 pgs e5776:4.6ds0) v4 ==== 751+0+0 (583122987 0 0) 0x22585640 con 0x1c5c2680 -1522> 2014-11-08 18:15:36.404924 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404841, event: header_read, op: pg_info(1 pgs e5776:4.6ds0) -1521> 2014-11-08 18:15:36.404929 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404843, event: throttled, op: pg_info(1 pgs e5776:4.6ds0) -1520> 2014-11-08 18:15:36.404933 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404895, event: all_read, op: pg_info(1 pgs e5776:4.6ds0) -1519> 2014-11-08 18:15:36.404936 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404922, event: dispatched, op: pg_info(1 pgs e5776:4.6ds0) -1518> 2014-11-08 18:15:36.404941 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404940, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.6ds0) -1517> 2014-11-08 18:15:36.404947 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404947, event: started, op: pg_info(1 pgs e5776:4.6ds0) -1516> 2014-11-08 18:15:36.404966 7fc5b9b29700 5 -- op tracker -- seq: 47323, time: 2014-11-08 18:15:36.404966, event: done, op: pg_info(1 pgs e5776:4.6ds0) -1515> 2014-11-08 18:15:36.405977 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12135 ==== pg_info(1 pgs e5776:4.cs1) v4 ==== 751+0+0 (1178725777 0 0) 0x225832a0 con 0x1c5c2680 -1514> 2014-11-08 18:15:36.405989 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.405933, event: header_read, op: pg_info(1 pgs e5776:4.cs1) -1513> 2014-11-08 18:15:36.405993 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.405934, event: throttled, op: pg_info(1 pgs e5776:4.cs1) -1512> 2014-11-08 18:15:36.405995 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.405967, event: all_read, op: pg_info(1 pgs e5776:4.cs1) -1511> 2014-11-08 18:15:36.405998 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.405987, event: dispatched, op: pg_info(1 pgs e5776:4.cs1) -1510> 2014-11-08 18:15:36.406001 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.406001, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.cs1) -1509> 2014-11-08 18:15:36.406005 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.406005, event: started, op: pg_info(1 pgs e5776:4.cs1) -1508> 2014-11-08 18:15:36.406018 7fc5b9b29700 5 -- op tracker -- seq: 47324, time: 2014-11-08 18:15:36.406018, event: done, op: pg_info(1 pgs e5776:4.cs1) -1507> 2014-11-08 18:15:36.406099 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.cs0) v4 -- ?+0 0x22d514a0 con 0x1c5c2680 -1506> 2014-11-08 18:15:36.406150 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5776 n=7607 ec=195 les/c 5776/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.921849 6 0.000326 -1505> 2014-11-08 18:15:36.406166 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5776 n=7607 ec=195 les/c 5776/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1504> 2014-11-08 18:15:36.406179 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5776 n=7607 ec=195 les/c 5776/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000014 0 0.000000 -1503> 2014-11-08 18:15:36.406188 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.cs0( v 5734'8057 (3458'5027,5734'8057] local-les=5776 n=7607 ec=195 les/c 5776/5525 5774/5774/5728) [9,1,2147483647] r=0 lpr=5774 pi=5524-5773/14 crt=5734'8041 lcod 5734'8056 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1502> 2014-11-08 18:15:36.406205 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.cs0) v4 -- ?+0 0x251581e0 con 0x1c5c2680 -1501> 2014-11-08 18:15:36.406367 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12136 ==== pg_info(1 pgs e5776:4.15s0) v4 ==== 771+0+0 (3686307276 0 0) 0x22583de0 con 0x1c5c2680 -1500> 2014-11-08 18:15:36.406377 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406327, event: header_read, op: pg_info(1 pgs e5776:4.15s0) -1499> 2014-11-08 18:15:36.406381 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406328, event: throttled, op: pg_info(1 pgs e5776:4.15s0) -1498> 2014-11-08 18:15:36.406383 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406357, event: all_read, op: pg_info(1 pgs e5776:4.15s0) -1497> 2014-11-08 18:15:36.406386 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406376, event: dispatched, op: pg_info(1 pgs e5776:4.15s0) -1496> 2014-11-08 18:15:36.406388 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406388, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.15s0) -1495> 2014-11-08 18:15:36.406395 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406394, event: started, op: pg_info(1 pgs e5776:4.15s0) -1494> 2014-11-08 18:15:36.406410 7fc5b9b29700 5 -- op tracker -- seq: 47325, time: 2014-11-08 18:15:36.406410, event: done, op: pg_info(1 pgs e5776:4.15s0) -1493> 2014-11-08 18:15:36.406654 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12137 ==== pg_info(1 pgs e5776:4.81s1) v4 ==== 751+0+0 (4109416207 0 0) 0x22587260 con 0x1c5c2680 -1492> 2014-11-08 18:15:36.406664 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406614, event: header_read, op: pg_info(1 pgs e5776:4.81s1) -1491> 2014-11-08 18:15:36.406669 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406615, event: throttled, op: pg_info(1 pgs e5776:4.81s1) -1490> 2014-11-08 18:15:36.406674 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406644, event: all_read, op: pg_info(1 pgs e5776:4.81s1) -1489> 2014-11-08 18:15:36.406677 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406662, event: dispatched, op: pg_info(1 pgs e5776:4.81s1) -1488> 2014-11-08 18:15:36.406681 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406681, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.81s1) -1487> 2014-11-08 18:15:36.406696 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406696, event: started, op: pg_info(1 pgs e5776:4.81s1) -1486> 2014-11-08 18:15:36.406711 7fc5b9b29700 5 -- op tracker -- seq: 47326, time: 2014-11-08 18:15:36.406711, event: done, op: pg_info(1 pgs e5776:4.81s1) -1485> 2014-11-08 18:15:36.406724 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.81s0) v4 -- ?+0 0x2515f260 con 0x1c5c2680 -1484> 2014-11-08 18:15:36.406788 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5776 n=7680 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.914314 6 0.000264 -1483> 2014-11-08 18:15:36.406801 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5776 n=7680 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1482> 2014-11-08 18:15:36.406813 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5776 n=7680 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000012 0 0.000000 -1481> 2014-11-08 18:15:36.406823 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.81s0( v 5734'8201 (3458'5200,5734'8201] local-les=5776 n=7680 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5665'8188 lcod 5734'8200 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1480> 2014-11-08 18:15:36.406864 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.81s0) v4 -- ?+0 0x2515fbc0 con 0x1c5c2680 -1479> 2014-11-08 18:15:36.407725 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12138 ==== pg_info(1 pgs e5776:4.75s1) v4 ==== 751+0+0 (883192631 0 0) 0x251581e0 con 0x1c5c2680 -1478> 2014-11-08 18:15:36.407736 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407635, event: header_read, op: pg_info(1 pgs e5776:4.75s1) -1477> 2014-11-08 18:15:36.407740 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407636, event: throttled, op: pg_info(1 pgs e5776:4.75s1) -1476> 2014-11-08 18:15:36.407745 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407676, event: all_read, op: pg_info(1 pgs e5776:4.75s1) -1475> 2014-11-08 18:15:36.407747 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407735, event: dispatched, op: pg_info(1 pgs e5776:4.75s1) -1474> 2014-11-08 18:15:36.407752 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407752, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.75s1) -1473> 2014-11-08 18:15:36.407757 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407757, event: started, op: pg_info(1 pgs e5776:4.75s1) -1472> 2014-11-08 18:15:36.407771 7fc5b9b29700 5 -- op tracker -- seq: 47327, time: 2014-11-08 18:15:36.407771, event: done, op: pg_info(1 pgs e5776:4.75s1) -1471> 2014-11-08 18:15:36.407780 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12139 ==== pg_info(1 pgs e5776:4.15s2) v4 ==== 751+0+0 (3364625628 0 0) 0x22d514a0 con 0x1c5c2680 -1470> 2014-11-08 18:15:36.407788 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407693, event: header_read, op: pg_info(1 pgs e5776:4.15s2) -1469> 2014-11-08 18:15:36.407793 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407694, event: throttled, op: pg_info(1 pgs e5776:4.15s2) -1468> 2014-11-08 18:15:36.407795 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407719, event: all_read, op: pg_info(1 pgs e5776:4.15s2) -1467> 2014-11-08 18:15:36.407799 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407786, event: dispatched, op: pg_info(1 pgs e5776:4.15s2) -1466> 2014-11-08 18:15:36.407803 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407803, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.15s2) -1465> 2014-11-08 18:15:36.407808 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407808, event: started, op: pg_info(1 pgs e5776:4.15s2) -1464> 2014-11-08 18:15:36.407820 7fc5b9b29700 5 -- op tracker -- seq: 47328, time: 2014-11-08 18:15:36.407819, event: done, op: pg_info(1 pgs e5776:4.15s2) -1463> 2014-11-08 18:15:36.407844 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.75s0) v4 -- ?+0 0x4534ec0 con 0x1c5c2680 -1462> 2014-11-08 18:15:36.407877 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5776 n=15343 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.914541 6 0.000398 -1461> 2014-11-08 18:15:36.407891 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5776 n=15343 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1460> 2014-11-08 18:15:36.407902 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5776 n=15343 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000011 0 0.000000 -1459> 2014-11-08 18:15:36.407913 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.75s0( v 5734'16481 (3473'13480,5734'16481] local-les=5776 n=15343 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'16472 lcod 5734'16480 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1458> 2014-11-08 18:15:36.407926 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.75s0) v4 -- ?+0 0x4530960 con 0x1c5c2680 -1457> 2014-11-08 18:15:36.408071 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12140 ==== pg_info(1 pgs e5776:4.40s1) v4 ==== 751+0+0 (3779101935 0 0) 0x1d325be0 con 0x1c5c2680 -1456> 2014-11-08 18:15:36.408081 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408035, event: header_read, op: pg_info(1 pgs e5776:4.40s1) -1455> 2014-11-08 18:15:36.408084 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408035, event: throttled, op: pg_info(1 pgs e5776:4.40s1) -1454> 2014-11-08 18:15:36.408121 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408062, event: all_read, op: pg_info(1 pgs e5776:4.40s1) -1453> 2014-11-08 18:15:36.408124 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408079, event: dispatched, op: pg_info(1 pgs e5776:4.40s1) -1452> 2014-11-08 18:15:36.408127 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408127, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.40s1) -1451> 2014-11-08 18:15:36.408131 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408131, event: started, op: pg_info(1 pgs e5776:4.40s1) -1450> 2014-11-08 18:15:36.408145 7fc5b9b29700 5 -- op tracker -- seq: 47329, time: 2014-11-08 18:15:36.408145, event: done, op: pg_info(1 pgs e5776:4.40s1) -1449> 2014-11-08 18:15:36.408164 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.40s0) v4 -- ?+0 0x4532580 con 0x1c5c2680 -1448> 2014-11-08 18:15:36.408207 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5776 n=15308 ec=195 les/c 5776/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.910111 6 0.000344 -1447> 2014-11-08 18:15:36.408220 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5776 n=15308 ec=195 les/c 5776/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1446> 2014-11-08 18:15:36.408231 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5776 n=15308 ec=195 les/c 5776/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000010 0 0.000000 -1445> 2014-11-08 18:15:36.408240 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.40s0( v 5734'16345 (3473'13233,5734'16345] local-les=5776 n=15308 ec=195 les/c 5776/5769 5774/5774/5768) [9,1,2147483647] r=0 lpr=5774 pi=5768-5773/1 crt=5734'16340 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1444> 2014-11-08 18:15:36.408254 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.40s0) v4 -- ?+0 0x4536360 con 0x1c5c2680 -1443> 2014-11-08 18:15:36.408395 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12141 ==== pg_info(1 pgs e5776:4.39s1) v4 ==== 751+0+0 (633835778 0 0) 0x1d321fe0 con 0x1c5c2680 -1442> 2014-11-08 18:15:36.408405 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408357, event: header_read, op: pg_info(1 pgs e5776:4.39s1) -1441> 2014-11-08 18:15:36.408409 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408358, event: throttled, op: pg_info(1 pgs e5776:4.39s1) -1440> 2014-11-08 18:15:36.408411 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408384, event: all_read, op: pg_info(1 pgs e5776:4.39s1) -1439> 2014-11-08 18:15:36.408413 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408403, event: dispatched, op: pg_info(1 pgs e5776:4.39s1) -1438> 2014-11-08 18:15:36.408416 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408416, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.39s1) -1437> 2014-11-08 18:15:36.408420 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408420, event: started, op: pg_info(1 pgs e5776:4.39s1) -1436> 2014-11-08 18:15:36.408432 7fc5b9b29700 5 -- op tracker -- seq: 47330, time: 2014-11-08 18:15:36.408432, event: done, op: pg_info(1 pgs e5776:4.39s1) -1435> 2014-11-08 18:15:36.408444 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.39s0) v4 -- ?+0 0x4532760 con 0x1c5c2680 -1434> 2014-11-08 18:15:36.408506 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5776 n=7739 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.909544 6 0.000851 -1433> 2014-11-08 18:15:36.408518 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5776 n=7739 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1432> 2014-11-08 18:15:36.408528 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5776 n=7739 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000009 0 0.000000 -1431> 2014-11-08 18:15:36.408535 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.39s0( v 5734'8297 (3458'5296,5734'8297] local-les=5776 n=7739 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8287 lcod 5734'8296 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1430> 2014-11-08 18:15:36.408546 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.39s0) v4 -- ?+0 0x4537da0 con 0x1c5c2680 -1429> 2014-11-08 18:15:36.408655 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12142 ==== pg_info(1 pgs e5776:4.9s1) v4 ==== 751+0+0 (51548839 0 0) 0x1d320b40 con 0x1c5c2680 -1428> 2014-11-08 18:15:36.408664 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408612, event: header_read, op: pg_info(1 pgs e5776:4.9s1) -1427> 2014-11-08 18:15:36.408668 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408613, event: throttled, op: pg_info(1 pgs e5776:4.9s1) -1426> 2014-11-08 18:15:36.408671 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408646, event: all_read, op: pg_info(1 pgs e5776:4.9s1) -1425> 2014-11-08 18:15:36.408673 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408663, event: dispatched, op: pg_info(1 pgs e5776:4.9s1) -1424> 2014-11-08 18:15:36.408676 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408676, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.9s1) -1423> 2014-11-08 18:15:36.408680 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408680, event: started, op: pg_info(1 pgs e5776:4.9s1) -1422> 2014-11-08 18:15:36.408700 7fc5b9b29700 5 -- op tracker -- seq: 47331, time: 2014-11-08 18:15:36.408700, event: done, op: pg_info(1 pgs e5776:4.9s1) -1421> 2014-11-08 18:15:36.408722 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.9s0) v4 -- ?+0 0x2515aee0 con 0x1c5c2680 -1420> 2014-11-08 18:15:36.408747 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12143 ==== pg_info(1 pgs e5776:4.31s2) v4 ==== 751+0+0 (840932872 0 0) 0x229912c0 con 0x1c5c2680 -1419> 2014-11-08 18:15:36.408755 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408656, event: header_read, op: pg_info(1 pgs e5776:4.31s2) -1418> 2014-11-08 18:15:36.408759 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408657, event: throttled, op: pg_info(1 pgs e5776:4.31s2) -1417> 2014-11-08 18:15:36.408752 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5776 n=7653 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.925045 6 0.000268 -1416> 2014-11-08 18:15:36.408762 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408738, event: all_read, op: pg_info(1 pgs e5776:4.31s2) -1415> 2014-11-08 18:15:36.408764 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408754, event: dispatched, op: pg_info(1 pgs e5776:4.31s2) -1414> 2014-11-08 18:15:36.408767 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408767, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.31s2) -1413> 2014-11-08 18:15:36.408763 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5776 n=7653 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1412> 2014-11-08 18:15:36.408771 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408771, event: started, op: pg_info(1 pgs e5776:4.31s2) -1411> 2014-11-08 18:15:36.408772 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5776 n=7653 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000009 0 0.000000 -1410> 2014-11-08 18:15:36.408782 7fc5b9b29700 5 -- op tracker -- seq: 47332, time: 2014-11-08 18:15:36.408782, event: done, op: pg_info(1 pgs e5776:4.31s2) -1409> 2014-11-08 18:15:36.408780 7fc5b1318700 5 osd.9 pg_epoch: 5776 pg[4.9s0( v 5734'8148 (3458'5145,5734'8148] local-les=5776 n=7653 ec=195 les/c 5776/5756 5774/5774/5729) [9,1,2147483647] r=0 lpr=5774 pi=5751-5773/1 crt=5700'8139 lcod 5734'8147 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1408> 2014-11-08 18:15:36.408791 7fc5b1318700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.9s0) v4 -- ?+0 0x22451c20 con 0x1c5c2680 -1407> 2014-11-08 18:15:36.408798 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.31s0) v4 -- ?+0 0x4533c00 con 0x1c5c2680 -1406> 2014-11-08 18:15:36.408822 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5776 n=7614 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.909348 6 0.000250 -1405> 2014-11-08 18:15:36.408832 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5776 n=7614 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1404> 2014-11-08 18:15:36.408840 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5776 n=7614 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000008 0 0.000000 -1403> 2014-11-08 18:15:36.408848 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.31s0( v 5734'8142 (3458'5141,5734'8142] local-les=5776 n=7614 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5734'8140 lcod 5734'8141 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1402> 2014-11-08 18:15:36.408902 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.31s0) v4 -- ?+0 0x2234a940 con 0x1c5c2680 -1401> 2014-11-08 18:15:36.409009 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12144 ==== pg_info(1 pgs e5776:4.es2) v4 ==== 751+0+0 (3247147275 0 0) 0x229910e0 con 0x1c5c2680 -1400> 2014-11-08 18:15:36.409017 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.408941, event: header_read, op: pg_info(1 pgs e5776:4.es2) -1399> 2014-11-08 18:15:36.409021 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.408942, event: throttled, op: pg_info(1 pgs e5776:4.es2) -1398> 2014-11-08 18:15:36.409024 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.408970, event: all_read, op: pg_info(1 pgs e5776:4.es2) -1397> 2014-11-08 18:15:36.409026 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.409016, event: dispatched, op: pg_info(1 pgs e5776:4.es2) -1396> 2014-11-08 18:15:36.409029 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.409029, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.es2) -1395> 2014-11-08 18:15:36.409032 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.409032, event: started, op: pg_info(1 pgs e5776:4.es2) -1394> 2014-11-08 18:15:36.409045 7fc5b9b29700 5 -- op tracker -- seq: 47333, time: 2014-11-08 18:15:36.409045, event: done, op: pg_info(1 pgs e5776:4.es2) -1393> 2014-11-08 18:15:36.409052 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12145 ==== pg_info(1 pgs e5776:4.36s0) v4 ==== 751+0+0 (1361035923 0 0) 0x22997440 con 0x1c5c2680 -1392> 2014-11-08 18:15:36.409058 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.408977, event: header_read, op: pg_info(1 pgs e5776:4.36s0) -1391> 2014-11-08 18:15:36.409061 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.408977, event: throttled, op: pg_info(1 pgs e5776:4.36s0) -1390> 2014-11-08 18:15:36.409063 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.409003, event: all_read, op: pg_info(1 pgs e5776:4.36s0) -1389> 2014-11-08 18:15:36.409065 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.409057, event: dispatched, op: pg_info(1 pgs e5776:4.36s0) -1388> 2014-11-08 18:15:36.409068 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.409068, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.36s0) -1387> 2014-11-08 18:15:36.409071 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.409071, event: started, op: pg_info(1 pgs e5776:4.36s0) -1386> 2014-11-08 18:15:36.409084 7fc5b9b29700 5 -- op tracker -- seq: 47334, time: 2014-11-08 18:15:36.409084, event: done, op: pg_info(1 pgs e5776:4.36s0) -1385> 2014-11-08 18:15:36.409127 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.es0) v4 -- ?+0 0x22d52b20 con 0x1c5c2680 -1384> 2014-11-08 18:15:36.409158 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5776 n=7729 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Activating 0.925189 6 0.000278 -1383> 2014-11-08 18:15:36.409168 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5776 n=7729 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Recovered -1382> 2014-11-08 18:15:36.409176 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5776 n=7729 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active+undersized+degraded] exit Started/Primary/Active/Recovered 0.000008 0 0.000000 -1381> 2014-11-08 18:15:36.409184 7fc5b1b19700 5 osd.9 pg_epoch: 5776 pg[4.es0( v 5734'8258 (3458'5257,5734'8258] local-les=5776 n=7729 ec=195 les/c 5776/5756 5774/5774/5729) [9,2147483647,1] r=0 lpr=5774 pi=5751-5773/1 crt=5394'8244 lcod 5734'8257 mlcod 0'0 active+undersized+degraded] enter Started/Primary/Active/Clean -1380> 2014-11-08 18:15:36.409195 7fc5b1b19700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_info(1 pgs e5776:4.es0) v4 -- ?+0 0x23c35820 con 0x1c5c2680 -1379> 2014-11-08 18:15:36.411289 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12146 ==== pg_info(1 pgs e5776:4.59s1) v4 ==== 751+0+0 (3010213254 0 0) 0x4537da0 con 0x1c5c2680 -1378> 2014-11-08 18:15:36.411304 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411243, event: header_read, op: pg_info(1 pgs e5776:4.59s1) -1377> 2014-11-08 18:15:36.411309 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411244, event: throttled, op: pg_info(1 pgs e5776:4.59s1) -1376> 2014-11-08 18:15:36.411311 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411276, event: all_read, op: pg_info(1 pgs e5776:4.59s1) -1375> 2014-11-08 18:15:36.411314 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411301, event: dispatched, op: pg_info(1 pgs e5776:4.59s1) -1374> 2014-11-08 18:15:36.411317 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411317, event: waiting_for_osdmap, op: pg_info(1 pgs e5776:4.59s1) -1373> 2014-11-08 18:15:36.411322 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411322, event: started, op: pg_info(1 pgs e5776:4.59s1) -1372> 2014-11-08 18:15:36.411337 7fc5b9b29700 5 -- op tracker -- seq: 47335, time: 2014-11-08 18:15:36.411337, event: done, op: pg_info(1 pgs e5776:4.59s1) -1371> 2014-11-08 18:15:36.416340 7fc5c362f700 5 osd.9 5776 tick -1370> 2014-11-08 18:15:36.550473 7fc5b0316700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6806/1411 -- pg_scan(digest 4.88s0 5661fd88/1000002b663.0000000d/head//4-MAX e 5776/5776) v2 -- ?+146 0x2287e300 con 0x1c5c2680 -1369> 2014-11-08 18:15:36.550511 7fc5b0316700 5 -- op tracker -- seq: 47294, time: 2014-11-08 18:15:36.550510, event: done, op: pg_scan(get_digest 4.88s1 5661fd88/1000002b663.0000000d/head//4-0//0//4 e 5775/5775) -1368> 2014-11-08 18:15:36.654586 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8653 ==== pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) v2 ==== 139+0+0 (1912984730 0 0) 0x22acbf00 con 0x1c5c0580 -1367> 2014-11-08 18:15:36.654634 7fc5a152e700 5 -- op tracker -- seq: 47336, time: 2014-11-08 18:15:36.654484, event: header_read, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1366> 2014-11-08 18:15:36.654644 7fc5a152e700 5 -- op tracker -- seq: 47336, time: 2014-11-08 18:15:36.654488, event: throttled, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1365> 2014-11-08 18:15:36.654649 7fc5a152e700 5 -- op tracker -- seq: 47336, time: 2014-11-08 18:15:36.654578, event: all_read, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1364> 2014-11-08 18:15:36.654654 7fc5a152e700 5 -- op tracker -- seq: 47336, time: 0.000000, event: dispatched, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1363> 2014-11-08 18:15:36.654712 7fc5ad310700 5 -- op tracker -- seq: 47336, time: 2014-11-08 18:15:36.654702, event: reached_pg, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1362> 2014-11-08 18:15:36.654724 7fc5ad310700 5 -- op tracker -- seq: 47336, time: 2014-11-08 18:15:36.654724, event: started, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1361> 2014-11-08 18:15:36.657526 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8654 ==== pg_notify(4.28s1(7),4.21s1(50) epoch 5775) v5 ==== 4861+0+0 (359297020 0 0) 0x4530000 con 0x1c5c0580 -1360> 2014-11-08 18:15:36.657543 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.656991, event: header_read, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1359> 2014-11-08 18:15:36.657548 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.656993, event: throttled, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1358> 2014-11-08 18:15:36.657551 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.657513, event: all_read, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1357> 2014-11-08 18:15:36.657554 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.657540, event: dispatched, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1356> 2014-11-08 18:15:36.657567 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.657567, event: waiting_for_osdmap, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1355> 2014-11-08 18:15:36.657573 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.657573, event: started, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1354> 2014-11-08 18:15:36.657610 7fc5b9b29700 5 -- op tracker -- seq: 47337, time: 2014-11-08 18:15:36.657610, event: done, op: pg_notify(4.28s1(7),4.21s1(50) epoch 5775) -1353> 2014-11-08 18:15:36.711331 7fc5ad310700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.15:6811/1606 -- pg_scan(digest 4.89s0 40520589/100000199b6.00000035/head//4-MAX e 5776/5776) v2 -- ?+430 0x22acba80 con 0x1c5c0580 -1352> 2014-11-08 18:15:36.711368 7fc5ad310700 5 -- op tracker -- seq: 47336, time: 2014-11-08 18:15:36.711368, event: done, op: pg_scan(get_digest 4.89s2 40520589/100000199b6.00000035/head//4-0//0//4 e 5774/5774) -1351> 2014-11-08 18:15:36.721404 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x25a070c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e50ad60).accept sd=135 192.168.1.16:39173/0 -1350> 2014-11-08 18:15:36.723531 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x1ea78b40 con 0x1e50ad60 -1349> 2014-11-08 18:15:36.723558 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1348> 2014-11-08 18:15:36.723897 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x22d51c20 con 0x1e50ad60 -1347> 2014-11-08 18:15:36.723909 7fc5b9b29700 5 -- op tracker -- seq: 47338, time: 2014-11-08 18:15:36.723862, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1346> 2014-11-08 18:15:36.723917 7fc5b9b29700 5 -- op tracker -- seq: 47338, time: 2014-11-08 18:15:36.723863, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1345> 2014-11-08 18:15:36.723920 7fc5b9b29700 5 -- op tracker -- seq: 47338, time: 2014-11-08 18:15:36.723887, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1344> 2014-11-08 18:15:36.723922 7fc5b9b29700 5 -- op tracker -- seq: 47338, time: 2014-11-08 18:15:36.723907, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1343> 2014-11-08 18:15:36.723925 7fc5b9b29700 5 -- op tracker -- seq: 47338, time: 2014-11-08 18:15:36.723925, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1342> 2014-11-08 18:15:36.723931 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1341> 2014-11-08 18:15:36.723935 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ad60 -- 0x25a070c0 -1340> 2014-11-08 18:15:36.723958 7fc5b9b29700 5 -- op tracker -- seq: 47338, time: 2014-11-08 18:15:36.723958, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1339> 2014-11-08 18:15:36.723975 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x25a070c0 sd=135 :6811 s=4 pgs=8233 cs=1 l=0 c=0x1e50ad60).reader couldn't read tag, (0) Success -1338> 2014-11-08 18:15:36.723994 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x25a070c0 sd=135 :6811 s=4 pgs=8233 cs=1 l=0 c=0x1e50ad60).fault (0) Success -1337> 2014-11-08 18:15:36.725067 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x25a06e00 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e509fa0).accept sd=135 192.168.1.16:39175/0 -1336> 2014-11-08 18:15:36.726027 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302de80 con 0x1e509fa0 -1335> 2014-11-08 18:15:36.726045 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1334> 2014-11-08 18:15:36.726106 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x22a21860 con 0x1e509fa0 -1333> 2014-11-08 18:15:36.726129 7fc5b9b29700 5 -- op tracker -- seq: 47339, time: 2014-11-08 18:15:36.726072, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1332> 2014-11-08 18:15:36.726137 7fc5b9b29700 5 -- op tracker -- seq: 47339, time: 2014-11-08 18:15:36.726073, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1331> 2014-11-08 18:15:36.726140 7fc5b9b29700 5 -- op tracker -- seq: 47339, time: 2014-11-08 18:15:36.726097, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1330> 2014-11-08 18:15:36.726144 7fc5b9b29700 5 -- op tracker -- seq: 47339, time: 2014-11-08 18:15:36.726127, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1329> 2014-11-08 18:15:36.726149 7fc5b9b29700 5 -- op tracker -- seq: 47339, time: 2014-11-08 18:15:36.726148, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1328> 2014-11-08 18:15:36.726159 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1327> 2014-11-08 18:15:36.726164 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509fa0 -- 0x25a06e00 -1326> 2014-11-08 18:15:36.726179 7fc5b9b29700 5 -- op tracker -- seq: 47339, time: 2014-11-08 18:15:36.726179, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1325> 2014-11-08 18:15:36.726192 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x25a06e00 sd=135 :6811 s=4 pgs=8235 cs=3 l=0 c=0x1e509fa0).reader couldn't read tag, (0) Success -1324> 2014-11-08 18:15:36.726203 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x25a06e00 sd=135 :6811 s=4 pgs=8235 cs=3 l=0 c=0x1e509fa0).fault (0) Success -1323> 2014-11-08 18:15:36.727143 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e77f0c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e509e40).accept sd=135 192.168.1.16:39176/0 -1322> 2014-11-08 18:15:36.729540 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302c800 con 0x1e509e40 -1321> 2014-11-08 18:15:36.729564 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1320> 2014-11-08 18:15:36.729956 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1f4c5280 con 0x1e509e40 -1319> 2014-11-08 18:15:36.729968 7fc5b9b29700 5 -- op tracker -- seq: 47340, time: 2014-11-08 18:15:36.729919, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1318> 2014-11-08 18:15:36.729976 7fc5b9b29700 5 -- op tracker -- seq: 47340, time: 2014-11-08 18:15:36.729920, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1317> 2014-11-08 18:15:36.729980 7fc5b9b29700 5 -- op tracker -- seq: 47340, time: 2014-11-08 18:15:36.729945, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1316> 2014-11-08 18:15:36.729983 7fc5b9b29700 5 -- op tracker -- seq: 47340, time: 2014-11-08 18:15:36.729966, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1315> 2014-11-08 18:15:36.729988 7fc5b9b29700 5 -- op tracker -- seq: 47340, time: 2014-11-08 18:15:36.729988, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1314> 2014-11-08 18:15:36.729994 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1313> 2014-11-08 18:15:36.729999 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509e40 -- 0x1e77f0c0 -1312> 2014-11-08 18:15:36.730071 7fc5b9b29700 5 -- op tracker -- seq: 47340, time: 2014-11-08 18:15:36.730071, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1311> 2014-11-08 18:15:36.730083 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1e897080 con 0x1e509e40 -1310> 2014-11-08 18:15:36.730092 7fc5b9b29700 5 -- op tracker -- seq: 47341, time: 2014-11-08 18:15:36.730010, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1309> 2014-11-08 18:15:36.730097 7fc5b9b29700 5 -- op tracker -- seq: 47341, time: 2014-11-08 18:15:36.730011, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1308> 2014-11-08 18:15:36.730101 7fc5b9b29700 5 -- op tracker -- seq: 47341, time: 2014-11-08 18:15:36.730031, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1307> 2014-11-08 18:15:36.730104 7fc5b9b29700 5 -- op tracker -- seq: 47341, time: 2014-11-08 18:15:36.730091, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1306> 2014-11-08 18:15:36.730108 7fc5b9b29700 5 -- op tracker -- seq: 47341, time: 2014-11-08 18:15:36.730108, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1305> 2014-11-08 18:15:36.730117 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1304> 2014-11-08 18:15:36.730121 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509e40 -- pipe dne -1303> 2014-11-08 18:15:36.730125 7fc5b9b29700 5 -- op tracker -- seq: 47341, time: 2014-11-08 18:15:36.730125, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1302> 2014-11-08 18:15:36.730133 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1e896720 con 0x1e509e40 -1301> 2014-11-08 18:15:36.730140 7fc5b9b29700 5 -- op tracker -- seq: 47342, time: 2014-11-08 18:15:36.730036, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1300> 2014-11-08 18:15:36.730145 7fc5b9b29700 5 -- op tracker -- seq: 47342, time: 2014-11-08 18:15:36.730036, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1299> 2014-11-08 18:15:36.730148 7fc5b9b29700 5 -- op tracker -- seq: 47342, time: 2014-11-08 18:15:36.730053, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1298> 2014-11-08 18:15:36.730152 7fc5b9b29700 5 -- op tracker -- seq: 47342, time: 2014-11-08 18:15:36.730139, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1297> 2014-11-08 18:15:36.730155 7fc5b9b29700 5 -- op tracker -- seq: 47342, time: 2014-11-08 18:15:36.730155, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1296> 2014-11-08 18:15:36.730159 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1295> 2014-11-08 18:15:36.730164 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509e40 -- pipe dne -1294> 2014-11-08 18:15:36.730167 7fc5b9b29700 5 -- op tracker -- seq: 47342, time: 2014-11-08 18:15:36.730167, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1293> 2014-11-08 18:15:36.730177 7fc5a27c0700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77f0c0 sd=135 :6811 s=4 pgs=8236 cs=5 l=0 c=0x1e509e40).do_sendmsg error (32) Broken pipe -1292> 2014-11-08 18:15:36.730197 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77f0c0 sd=135 :6811 s=4 pgs=8236 cs=5 l=0 c=0x1e509e40).writer couldn't write ack, (32) Broken pipe -1291> 2014-11-08 18:15:36.730206 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77f0c0 sd=135 :6811 s=4 pgs=8236 cs=5 l=0 c=0x1e509e40).fault (32) Broken pipe -1290> 2014-11-08 18:15:36.730242 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77f0c0 sd=135 :6811 s=4 pgs=8236 cs=5 l=0 c=0x1e509e40).reader couldn't read tag, (0) Success -1289> 2014-11-08 18:15:36.730253 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77f0c0 sd=135 :6811 s=4 pgs=8236 cs=5 l=0 c=0x1e509e40).fault (0) Success -1288> 2014-11-08 18:15:36.731648 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e778000 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e509ce0).accept sd=135 192.168.1.16:39177/0 -1287> 2014-11-08 18:15:36.736289 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302e540 con 0x1e509ce0 -1286> 2014-11-08 18:15:36.736321 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1285> 2014-11-08 18:15:36.736380 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1e8932a0 con 0x1e509ce0 -1284> 2014-11-08 18:15:36.736408 7fc5b9b29700 5 -- op tracker -- seq: 47343, time: 2014-11-08 18:15:36.736335, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1283> 2014-11-08 18:15:36.736415 7fc5b9b29700 5 -- op tracker -- seq: 47343, time: 2014-11-08 18:15:36.736336, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1282> 2014-11-08 18:15:36.736418 7fc5b9b29700 5 -- op tracker -- seq: 47343, time: 2014-11-08 18:15:36.736369, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1281> 2014-11-08 18:15:36.736420 7fc5b9b29700 5 -- op tracker -- seq: 47343, time: 2014-11-08 18:15:36.736390, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1280> 2014-11-08 18:15:36.736423 7fc5b9b29700 5 -- op tracker -- seq: 47343, time: 2014-11-08 18:15:36.736423, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1279> 2014-11-08 18:15:36.736428 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1278> 2014-11-08 18:15:36.736433 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509ce0 -- 0x1e778000 -1277> 2014-11-08 18:15:36.736465 7fc5b9b29700 5 -- op tracker -- seq: 47343, time: 2014-11-08 18:15:36.736465, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1276> 2014-11-08 18:15:36.736481 7fc5a27c0700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778000 sd=135 :6811 s=4 pgs=8237 cs=7 l=0 c=0x1e509ce0).do_sendmsg error (32) Broken pipe -1275> 2014-11-08 18:15:36.736509 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778000 sd=135 :6811 s=4 pgs=8237 cs=7 l=0 c=0x1e509ce0).writer couldn't write ack, (32) Broken pipe -1274> 2014-11-08 18:15:36.736518 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778000 sd=135 :6811 s=4 pgs=8237 cs=7 l=0 c=0x1e509ce0).fault (32) Broken pipe -1273> 2014-11-08 18:15:36.738962 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e778840 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e50ef60).accept sd=135 192.168.1.16:39178/0 -1272> 2014-11-08 18:15:36.743357 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302c140 con 0x1e50ef60 -1271> 2014-11-08 18:15:36.743379 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1270> 2014-11-08 18:15:36.744179 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x227cecc0 con 0x1e50ef60 -1269> 2014-11-08 18:15:36.744192 7fc5b9b29700 5 -- op tracker -- seq: 47344, time: 2014-11-08 18:15:36.744128, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1268> 2014-11-08 18:15:36.744199 7fc5b9b29700 5 -- op tracker -- seq: 47344, time: 2014-11-08 18:15:36.744129, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1267> 2014-11-08 18:15:36.744202 7fc5b9b29700 5 -- op tracker -- seq: 47344, time: 2014-11-08 18:15:36.744169, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1266> 2014-11-08 18:15:36.744204 7fc5b9b29700 5 -- op tracker -- seq: 47344, time: 2014-11-08 18:15:36.744190, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1265> 2014-11-08 18:15:36.744207 7fc5b9b29700 5 -- op tracker -- seq: 47344, time: 2014-11-08 18:15:36.744207, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1264> 2014-11-08 18:15:36.744212 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1263> 2014-11-08 18:15:36.744216 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ef60 -- 0x1e778840 -1262> 2014-11-08 18:15:36.744305 7fc5b9b29700 5 -- op tracker -- seq: 47344, time: 2014-11-08 18:15:36.744305, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1261> 2014-11-08 18:15:36.744318 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x227c8d20 con 0x1e50ef60 -1260> 2014-11-08 18:15:36.744325 7fc5b9b29700 5 -- op tracker -- seq: 47345, time: 2014-11-08 18:15:36.744231, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1259> 2014-11-08 18:15:36.744328 7fc5b9b29700 5 -- op tracker -- seq: 47345, time: 2014-11-08 18:15:36.744231, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1258> 2014-11-08 18:15:36.744330 7fc5b9b29700 5 -- op tracker -- seq: 47345, time: 2014-11-08 18:15:36.744264, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1257> 2014-11-08 18:15:36.744333 7fc5b9b29700 5 -- op tracker -- seq: 47345, time: 2014-11-08 18:15:36.744324, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1256> 2014-11-08 18:15:36.744335 7fc5b9b29700 5 -- op tracker -- seq: 47345, time: 2014-11-08 18:15:36.744335, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1255> 2014-11-08 18:15:36.744338 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1254> 2014-11-08 18:15:36.744341 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ef60 -- pipe dne -1253> 2014-11-08 18:15:36.744344 7fc5b9b29700 5 -- op tracker -- seq: 47345, time: 2014-11-08 18:15:36.744344, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1252> 2014-11-08 18:15:36.744349 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x227cc560 con 0x1e50ef60 -1251> 2014-11-08 18:15:36.744354 7fc5b9b29700 5 -- op tracker -- seq: 47346, time: 2014-11-08 18:15:36.744269, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1250> 2014-11-08 18:15:36.744357 7fc5b9b29700 5 -- op tracker -- seq: 47346, time: 2014-11-08 18:15:36.744269, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1249> 2014-11-08 18:15:36.744359 7fc5b9b29700 5 -- op tracker -- seq: 47346, time: 2014-11-08 18:15:36.744287, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1248> 2014-11-08 18:15:36.744362 7fc5b9b29700 5 -- op tracker -- seq: 47346, time: 2014-11-08 18:15:36.744353, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1247> 2014-11-08 18:15:36.744365 7fc5b9b29700 5 -- op tracker -- seq: 47346, time: 2014-11-08 18:15:36.744365, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1246> 2014-11-08 18:15:36.744367 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1245> 2014-11-08 18:15:36.744370 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ef60 -- pipe dne -1244> 2014-11-08 18:15:36.744372 7fc5b9b29700 5 -- op tracker -- seq: 47346, time: 2014-11-08 18:15:36.744372, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1243> 2014-11-08 18:15:36.744390 7fc5a27c0700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778840 sd=135 :6811 s=4 pgs=8238 cs=9 l=0 c=0x1e50ef60).do_sendmsg error (32) Broken pipe -1242> 2014-11-08 18:15:36.744410 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778840 sd=135 :6811 s=4 pgs=8238 cs=9 l=0 c=0x1e50ef60).writer couldn't write ack, (32) Broken pipe -1241> 2014-11-08 18:15:36.744419 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778840 sd=135 :6811 s=4 pgs=8238 cs=9 l=0 c=0x1e50ef60).fault (32) Broken pipe -1240> 2014-11-08 18:15:36.744470 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778840 sd=135 :6811 s=4 pgs=8238 cs=9 l=0 c=0x1e50ef60).reader couldn't read tag, (0) Success -1239> 2014-11-08 18:15:36.744480 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778840 sd=135 :6811 s=4 pgs=8238 cs=9 l=0 c=0x1e50ef60).fault (0) Success -1238> 2014-11-08 18:15:36.747645 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e77e880 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e50c620).accept sd=135 192.168.1.16:39179/0 -1237> 2014-11-08 18:15:36.750545 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302ee40 con 0x1e50c620 -1236> 2014-11-08 18:15:36.750564 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1235> 2014-11-08 18:15:36.751161 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x227c92c0 con 0x1e50c620 -1234> 2014-11-08 18:15:36.751173 7fc5b9b29700 5 -- op tracker -- seq: 47347, time: 2014-11-08 18:15:36.751119, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1233> 2014-11-08 18:15:36.751180 7fc5b9b29700 5 -- op tracker -- seq: 47347, time: 2014-11-08 18:15:36.751120, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1232> 2014-11-08 18:15:36.751185 7fc5b9b29700 5 -- op tracker -- seq: 47347, time: 2014-11-08 18:15:36.751144, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1231> 2014-11-08 18:15:36.751189 7fc5b9b29700 5 -- op tracker -- seq: 47347, time: 2014-11-08 18:15:36.751171, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1230> 2014-11-08 18:15:36.751192 7fc5b9b29700 5 -- op tracker -- seq: 47347, time: 2014-11-08 18:15:36.751192, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1229> 2014-11-08 18:15:36.751197 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1228> 2014-11-08 18:15:36.751201 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50c620 -- 0x1e77e880 -1227> 2014-11-08 18:15:36.751210 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77e880 sd=135 :6811 s=4 pgs=8239 cs=11 l=0 c=0x1e50c620).fault (0) Success -1226> 2014-11-08 18:15:36.751213 7fc5b9b29700 5 -- op tracker -- seq: 47347, time: 2014-11-08 18:15:36.751213, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1225> 2014-11-08 18:15:36.751298 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1dbf9c20 con 0x1e50c620 -1224> 2014-11-08 18:15:36.751305 7fc5b9b29700 5 -- op tracker -- seq: 47348, time: 2014-11-08 18:15:36.751152, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1223> 2014-11-08 18:15:36.751308 7fc5b9b29700 5 -- op tracker -- seq: 47348, time: 2014-11-08 18:15:36.751152, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1222> 2014-11-08 18:15:36.751311 7fc5b9b29700 5 -- op tracker -- seq: 47348, time: 2014-11-08 18:15:36.751177, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1221> 2014-11-08 18:15:36.751314 7fc5b9b29700 5 -- op tracker -- seq: 47348, time: 2014-11-08 18:15:36.751304, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1220> 2014-11-08 18:15:36.751318 7fc5b9b29700 5 -- op tracker -- seq: 47348, time: 2014-11-08 18:15:36.751318, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1219> 2014-11-08 18:15:36.751323 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1218> 2014-11-08 18:15:36.751328 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50c620 -- pipe dne -1217> 2014-11-08 18:15:36.751332 7fc5b9b29700 5 -- op tracker -- seq: 47348, time: 2014-11-08 18:15:36.751332, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1216> 2014-11-08 18:15:36.754232 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e77cfc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0dc0).accept sd=135 192.168.1.16:39180/0 -1215> 2014-11-08 18:15:36.758936 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23029d40 con 0x23bb0dc0 -1214> 2014-11-08 18:15:36.758958 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1213> 2014-11-08 18:15:36.761829 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x227cdbe0 con 0x23bb0dc0 -1212> 2014-11-08 18:15:36.761851 7fc5b9b29700 5 -- op tracker -- seq: 47349, time: 2014-11-08 18:15:36.761708, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1211> 2014-11-08 18:15:36.761858 7fc5b9b29700 5 -- op tracker -- seq: 47349, time: 2014-11-08 18:15:36.761711, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1210> 2014-11-08 18:15:36.761861 7fc5b9b29700 5 -- op tracker -- seq: 47349, time: 2014-11-08 18:15:36.761763, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1209> 2014-11-08 18:15:36.761863 7fc5b9b29700 5 -- op tracker -- seq: 47349, time: 2014-11-08 18:15:36.761848, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1208> 2014-11-08 18:15:36.761867 7fc5b9b29700 5 -- op tracker -- seq: 47349, time: 2014-11-08 18:15:36.761867, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1207> 2014-11-08 18:15:36.761872 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1206> 2014-11-08 18:15:36.761876 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0dc0 -- 0x1e77cfc0 -1205> 2014-11-08 18:15:36.761898 7fc5b9b29700 5 -- op tracker -- seq: 47349, time: 2014-11-08 18:15:36.761898, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1204> 2014-11-08 18:15:36.761910 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e0f00 con 0x23bb0dc0 -1203> 2014-11-08 18:15:36.761916 7fc5b9b29700 5 -- op tracker -- seq: 47350, time: 2014-11-08 18:15:36.761775, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1202> 2014-11-08 18:15:36.761919 7fc5b9b29700 5 -- op tracker -- seq: 47350, time: 2014-11-08 18:15:36.761776, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1201> 2014-11-08 18:15:36.761921 7fc5b9b29700 5 -- op tracker -- seq: 47350, time: 2014-11-08 18:15:36.761795, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1200> 2014-11-08 18:15:36.761923 7fc5b9b29700 5 -- op tracker -- seq: 47350, time: 2014-11-08 18:15:36.761915, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1199> 2014-11-08 18:15:36.761926 7fc5b9b29700 5 -- op tracker -- seq: 47350, time: 2014-11-08 18:15:36.761926, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1198> 2014-11-08 18:15:36.761929 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1197> 2014-11-08 18:15:36.761932 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0dc0 -- pipe dne -1196> 2014-11-08 18:15:36.761934 7fc5b9b29700 5 -- op tracker -- seq: 47350, time: 2014-11-08 18:15:36.761934, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1195> 2014-11-08 18:15:36.761940 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e1a40 con 0x23bb0dc0 -1194> 2014-11-08 18:15:36.761945 7fc5b9b29700 5 -- op tracker -- seq: 47351, time: 2014-11-08 18:15:36.761800, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1193> 2014-11-08 18:15:36.761948 7fc5b9b29700 5 -- op tracker -- seq: 47351, time: 2014-11-08 18:15:36.761800, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1192> 2014-11-08 18:15:36.761950 7fc5b9b29700 5 -- op tracker -- seq: 47351, time: 2014-11-08 18:15:36.761819, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1191> 2014-11-08 18:15:36.761953 7fc5b9b29700 5 -- op tracker -- seq: 47351, time: 2014-11-08 18:15:36.761944, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1190> 2014-11-08 18:15:36.761955 7fc5b9b29700 5 -- op tracker -- seq: 47351, time: 2014-11-08 18:15:36.761955, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1189> 2014-11-08 18:15:36.761958 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1188> 2014-11-08 18:15:36.761961 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0dc0 -- pipe dne -1187> 2014-11-08 18:15:36.761963 7fc5b9b29700 5 -- op tracker -- seq: 47351, time: 2014-11-08 18:15:36.761963, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1186> 2014-11-08 18:15:36.762033 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77cfc0 sd=135 :6811 s=4 pgs=8240 cs=13 l=0 c=0x23bb0dc0).reader couldn't read tag, (0) Success -1185> 2014-11-08 18:15:36.762049 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77cfc0 sd=135 :6811 s=4 pgs=8240 cs=13 l=0 c=0x23bb0dc0).fault (0) Success -1184> 2014-11-08 18:15:36.766103 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e77fbc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0000).accept sd=135 192.168.1.16:39181/0 -1183> 2014-11-08 18:15:36.770591 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23029f80 con 0x23bb0000 -1182> 2014-11-08 18:15:36.770620 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1181> 2014-11-08 18:15:36.770819 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e6180 con 0x23bb0000 -1180> 2014-11-08 18:15:36.770832 7fc5b9b29700 5 -- op tracker -- seq: 47352, time: 2014-11-08 18:15:36.770766, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1179> 2014-11-08 18:15:36.770838 7fc5b9b29700 5 -- op tracker -- seq: 47352, time: 2014-11-08 18:15:36.770767, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1178> 2014-11-08 18:15:36.770840 7fc5b9b29700 5 -- op tracker -- seq: 47352, time: 2014-11-08 18:15:36.770792, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1177> 2014-11-08 18:15:36.770843 7fc5b9b29700 5 -- op tracker -- seq: 47352, time: 2014-11-08 18:15:36.770830, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1176> 2014-11-08 18:15:36.770845 7fc5b9b29700 5 -- op tracker -- seq: 47352, time: 2014-11-08 18:15:36.770845, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1175> 2014-11-08 18:15:36.770851 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1174> 2014-11-08 18:15:36.770855 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0000 -- 0x1e77fbc0 -1173> 2014-11-08 18:15:36.770878 7fc5b9b29700 5 -- op tracker -- seq: 47352, time: 2014-11-08 18:15:36.770878, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1172> 2014-11-08 18:15:36.770894 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77fbc0 sd=135 :6811 s=4 pgs=8241 cs=15 l=0 c=0x23bb0000).reader couldn't read tag, (0) Success -1171> 2014-11-08 18:15:36.770923 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77fbc0 sd=135 :6811 s=4 pgs=8241 cs=15 l=0 c=0x23bb0000).fault (0) Success -1170> 2014-11-08 18:15:36.771760 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e77e040 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb39c0).accept sd=135 192.168.1.16:39182/0 -1169> 2014-11-08 18:15:36.775243 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302d7c0 con 0x23bb39c0 -1168> 2014-11-08 18:15:36.775261 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1167> 2014-11-08 18:15:36.775375 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e6720 con 0x23bb39c0 -1166> 2014-11-08 18:15:36.775390 7fc5b9b29700 5 -- op tracker -- seq: 47353, time: 2014-11-08 18:15:36.775341, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1165> 2014-11-08 18:15:36.775397 7fc5b9b29700 5 -- op tracker -- seq: 47353, time: 2014-11-08 18:15:36.775342, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1164> 2014-11-08 18:15:36.775400 7fc5b9b29700 5 -- op tracker -- seq: 47353, time: 2014-11-08 18:15:36.775365, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1163> 2014-11-08 18:15:36.775404 7fc5b9b29700 5 -- op tracker -- seq: 47353, time: 2014-11-08 18:15:36.775388, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1162> 2014-11-08 18:15:36.775407 7fc5b9b29700 5 -- op tracker -- seq: 47353, time: 2014-11-08 18:15:36.775407, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1161> 2014-11-08 18:15:36.775413 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1160> 2014-11-08 18:15:36.775418 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb39c0 -- 0x1e77e040 -1159> 2014-11-08 18:15:36.775488 7fc5b9b29700 5 -- op tracker -- seq: 47353, time: 2014-11-08 18:15:36.775488, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1158> 2014-11-08 18:15:36.775499 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e6cc0 con 0x23bb39c0 -1157> 2014-11-08 18:15:36.775508 7fc5b9b29700 5 -- op tracker -- seq: 47354, time: 2014-11-08 18:15:36.775427, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1156> 2014-11-08 18:15:36.775512 7fc5b9b29700 5 -- op tracker -- seq: 47354, time: 2014-11-08 18:15:36.775428, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1155> 2014-11-08 18:15:36.775516 7fc5b9b29700 5 -- op tracker -- seq: 47354, time: 2014-11-08 18:15:36.775448, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1154> 2014-11-08 18:15:36.775519 7fc5b9b29700 5 -- op tracker -- seq: 47354, time: 2014-11-08 18:15:36.775507, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1153> 2014-11-08 18:15:36.775522 7fc5b9b29700 5 -- op tracker -- seq: 47354, time: 2014-11-08 18:15:36.775521, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1152> 2014-11-08 18:15:36.775526 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1151> 2014-11-08 18:15:36.775530 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb39c0 -- pipe dne -1150> 2014-11-08 18:15:36.775532 7fc5b9b29700 5 -- op tracker -- seq: 47354, time: 2014-11-08 18:15:36.775532, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1149> 2014-11-08 18:15:36.775540 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e50a0 con 0x23bb39c0 -1148> 2014-11-08 18:15:36.775547 7fc5b9b29700 5 -- op tracker -- seq: 47355, time: 2014-11-08 18:15:36.775453, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1147> 2014-11-08 18:15:36.775552 7fc5b9b29700 5 -- op tracker -- seq: 47355, time: 2014-11-08 18:15:36.775454, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1146> 2014-11-08 18:15:36.775557 7fc5b9b29700 5 -- op tracker -- seq: 47355, time: 2014-11-08 18:15:36.775471, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1145> 2014-11-08 18:15:36.775560 7fc5b9b29700 5 -- op tracker -- seq: 47355, time: 2014-11-08 18:15:36.775547, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1144> 2014-11-08 18:15:36.775564 7fc5b9b29700 5 -- op tracker -- seq: 47355, time: 2014-11-08 18:15:36.775564, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1143> 2014-11-08 18:15:36.775567 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1142> 2014-11-08 18:15:36.775572 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb39c0 -- pipe dne -1141> 2014-11-08 18:15:36.775575 7fc5b9b29700 5 -- op tracker -- seq: 47355, time: 2014-11-08 18:15:36.775575, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1140> 2014-11-08 18:15:36.775586 7fc5a27c0700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77e040 sd=135 :6811 s=4 pgs=8242 cs=17 l=0 c=0x23bb39c0).do_sendmsg error (32) Broken pipe -1139> 2014-11-08 18:15:36.775601 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77e040 sd=135 :6811 s=4 pgs=8242 cs=17 l=0 c=0x23bb39c0).writer couldn't write ack, (32) Broken pipe -1138> 2014-11-08 18:15:36.775610 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77e040 sd=135 :6811 s=4 pgs=8242 cs=17 l=0 c=0x23bb39c0).fault (32) Broken pipe -1137> 2014-11-08 18:15:36.775641 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77e040 sd=135 :6811 s=4 pgs=8242 cs=17 l=0 c=0x23bb39c0).reader couldn't read tag, (0) Success -1136> 2014-11-08 18:15:36.775651 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e77e040 sd=135 :6811 s=4 pgs=8242 cs=17 l=0 c=0x23bb39c0).fault (0) Success -1135> 2014-11-08 18:15:36.780074 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1e778b00 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0160).accept sd=135 192.168.1.16:39183/0 -1134> 2014-11-08 18:15:36.787972 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302f500 con 0x23bb0160 -1133> 2014-11-08 18:15:36.787996 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1132> 2014-11-08 18:15:36.790029 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e1680 con 0x23bb0160 -1131> 2014-11-08 18:15:36.790050 7fc5b9b29700 5 -- op tracker -- seq: 47356, time: 2014-11-08 18:15:36.789912, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1130> 2014-11-08 18:15:36.790057 7fc5b9b29700 5 -- op tracker -- seq: 47356, time: 2014-11-08 18:15:36.789915, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1129> 2014-11-08 18:15:36.790060 7fc5b9b29700 5 -- op tracker -- seq: 47356, time: 2014-11-08 18:15:36.789966, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1128> 2014-11-08 18:15:36.790062 7fc5b9b29700 5 -- op tracker -- seq: 47356, time: 2014-11-08 18:15:36.790046, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1127> 2014-11-08 18:15:36.790065 7fc5b9b29700 5 -- op tracker -- seq: 47356, time: 2014-11-08 18:15:36.790065, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1126> 2014-11-08 18:15:36.790070 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1125> 2014-11-08 18:15:36.790075 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0160 -- 0x1e778b00 -1124> 2014-11-08 18:15:36.790095 7fc5b9b29700 5 -- op tracker -- seq: 47356, time: 2014-11-08 18:15:36.790095, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1123> 2014-11-08 18:15:36.790107 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e6360 con 0x23bb0160 -1122> 2014-11-08 18:15:36.790113 7fc5b9b29700 5 -- op tracker -- seq: 47357, time: 2014-11-08 18:15:36.789978, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1121> 2014-11-08 18:15:36.790116 7fc5b9b29700 5 -- op tracker -- seq: 47357, time: 2014-11-08 18:15:36.789978, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1120> 2014-11-08 18:15:36.790118 7fc5b9b29700 5 -- op tracker -- seq: 47357, time: 2014-11-08 18:15:36.789997, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1119> 2014-11-08 18:15:36.790120 7fc5b9b29700 5 -- op tracker -- seq: 47357, time: 2014-11-08 18:15:36.790112, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1118> 2014-11-08 18:15:36.790124 7fc5b9b29700 5 -- op tracker -- seq: 47357, time: 2014-11-08 18:15:36.790123, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1117> 2014-11-08 18:15:36.790127 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1116> 2014-11-08 18:15:36.790130 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0160 -- pipe dne -1115> 2014-11-08 18:15:36.790137 7fc5b9b29700 5 -- op tracker -- seq: 47357, time: 2014-11-08 18:15:36.790136, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1114> 2014-11-08 18:15:36.790142 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e7bc0 con 0x23bb0160 -1113> 2014-11-08 18:15:36.790148 7fc5b9b29700 5 -- op tracker -- seq: 47358, time: 2014-11-08 18:15:36.790002, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1112> 2014-11-08 18:15:36.790151 7fc5b9b29700 5 -- op tracker -- seq: 47358, time: 2014-11-08 18:15:36.790002, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1111> 2014-11-08 18:15:36.790153 7fc5b9b29700 5 -- op tracker -- seq: 47358, time: 2014-11-08 18:15:36.790020, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1110> 2014-11-08 18:15:36.790155 7fc5b9b29700 5 -- op tracker -- seq: 47358, time: 2014-11-08 18:15:36.790147, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1109> 2014-11-08 18:15:36.790158 7fc5b9b29700 5 -- op tracker -- seq: 47358, time: 2014-11-08 18:15:36.790158, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1108> 2014-11-08 18:15:36.790161 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1107> 2014-11-08 18:15:36.790164 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0160 -- pipe dne -1106> 2014-11-08 18:15:36.790166 7fc5b9b29700 5 -- op tracker -- seq: 47358, time: 2014-11-08 18:15:36.790166, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1105> 2014-11-08 18:15:36.790189 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778b00 sd=135 :6811 s=4 pgs=8243 cs=19 l=0 c=0x23bb0160).reader couldn't read tag, (0) Success -1104> 2014-11-08 18:15:36.790206 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1e778b00 sd=135 :6811 s=4 pgs=8243 cs=19 l=0 c=0x23bb0160).fault (0) Success -1103> 2014-11-08 18:15:36.796158 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d22b9c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb5d80).accept sd=135 192.168.1.16:39184/0 -1102> 2014-11-08 18:15:36.804175 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302ad00 con 0x23bb5d80 -1101> 2014-11-08 18:15:36.804198 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1100> 2014-11-08 18:15:36.809835 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e7da0 con 0x23bb5d80 -1099> 2014-11-08 18:15:36.809859 7fc5b9b29700 5 -- op tracker -- seq: 47359, time: 2014-11-08 18:15:36.809754, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1098> 2014-11-08 18:15:36.809866 7fc5b9b29700 5 -- op tracker -- seq: 47359, time: 2014-11-08 18:15:36.809757, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1097> 2014-11-08 18:15:36.809869 7fc5b9b29700 5 -- op tracker -- seq: 47359, time: 2014-11-08 18:15:36.809819, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1096> 2014-11-08 18:15:36.809872 7fc5b9b29700 5 -- op tracker -- seq: 47359, time: 2014-11-08 18:15:36.809855, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1095> 2014-11-08 18:15:36.809875 7fc5b9b29700 5 -- op tracker -- seq: 47359, time: 2014-11-08 18:15:36.809875, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1094> 2014-11-08 18:15:36.809880 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1093> 2014-11-08 18:15:36.809885 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb5d80 -- 0x1d22b9c0 -1092> 2014-11-08 18:15:36.809903 7fc5b9b29700 5 -- op tracker -- seq: 47359, time: 2014-11-08 18:15:36.809903, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1091> 2014-11-08 18:15:36.809916 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1d06eae0 con 0x23bb5d80 -1090> 2014-11-08 18:15:36.809922 7fc5b9b29700 5 -- op tracker -- seq: 47360, time: 2014-11-08 18:15:36.809835, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1089> 2014-11-08 18:15:36.809925 7fc5b9b29700 5 -- op tracker -- seq: 47360, time: 2014-11-08 18:15:36.809836, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1088> 2014-11-08 18:15:36.809927 7fc5b9b29700 5 -- op tracker -- seq: 47360, time: 2014-11-08 18:15:36.809856, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1087> 2014-11-08 18:15:36.809930 7fc5b9b29700 5 -- op tracker -- seq: 47360, time: 2014-11-08 18:15:36.809921, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1086> 2014-11-08 18:15:36.809932 7fc5b9b29700 5 -- op tracker -- seq: 47360, time: 2014-11-08 18:15:36.809932, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1085> 2014-11-08 18:15:36.809935 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1084> 2014-11-08 18:15:36.809938 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb5d80 -- pipe dne -1083> 2014-11-08 18:15:36.809941 7fc5b9b29700 5 -- op tracker -- seq: 47360, time: 2014-11-08 18:15:36.809941, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1082> 2014-11-08 18:15:36.809946 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x234e3de0 con 0x23bb5d80 -1081> 2014-11-08 18:15:36.809952 7fc5b9b29700 5 -- op tracker -- seq: 47361, time: 2014-11-08 18:15:36.809861, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1080> 2014-11-08 18:15:36.809955 7fc5b9b29700 5 -- op tracker -- seq: 47361, time: 2014-11-08 18:15:36.809861, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1079> 2014-11-08 18:15:36.809957 7fc5b9b29700 5 -- op tracker -- seq: 47361, time: 2014-11-08 18:15:36.809881, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1078> 2014-11-08 18:15:36.809960 7fc5b9b29700 5 -- op tracker -- seq: 47361, time: 2014-11-08 18:15:36.809951, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1077> 2014-11-08 18:15:36.809962 7fc5b9b29700 5 -- op tracker -- seq: 47361, time: 2014-11-08 18:15:36.809962, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1076> 2014-11-08 18:15:36.809965 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1075> 2014-11-08 18:15:36.809969 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb5d80 -- pipe dne -1074> 2014-11-08 18:15:36.809971 7fc5b9b29700 5 -- op tracker -- seq: 47361, time: 2014-11-08 18:15:36.809971, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1073> 2014-11-08 18:15:36.809997 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d22b9c0 sd=135 :6811 s=4 pgs=8244 cs=21 l=0 c=0x23bb5d80).reader couldn't read tag, (0) Success -1072> 2014-11-08 18:15:36.810015 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d22b9c0 sd=135 :6811 s=4 pgs=8244 cs=21 l=0 c=0x23bb5d80).fault (0) Success -1071> 2014-11-08 18:15:36.816099 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d39b180 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb1600).accept sd=135 192.168.1.16:39185/0 -1070> 2014-11-08 18:15:36.825425 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302a640 con 0x23bb1600 -1069> 2014-11-08 18:15:36.825459 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1068> 2014-11-08 18:15:36.828301 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1d06a1c0 con 0x23bb1600 -1067> 2014-11-08 18:15:36.828315 7fc5b9b29700 5 -- op tracker -- seq: 47362, time: 2014-11-08 18:15:36.828247, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1066> 2014-11-08 18:15:36.828323 7fc5b9b29700 5 -- op tracker -- seq: 47362, time: 2014-11-08 18:15:36.828249, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1065> 2014-11-08 18:15:36.828325 7fc5b9b29700 5 -- op tracker -- seq: 47362, time: 2014-11-08 18:15:36.828291, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1064> 2014-11-08 18:15:36.828330 7fc5b9b29700 5 -- op tracker -- seq: 47362, time: 2014-11-08 18:15:36.828313, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1063> 2014-11-08 18:15:36.828335 7fc5b9b29700 5 -- op tracker -- seq: 47362, time: 2014-11-08 18:15:36.828335, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1062> 2014-11-08 18:15:36.828342 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1061> 2014-11-08 18:15:36.828348 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb1600 -- 0x1d39b180 -1060> 2014-11-08 18:15:36.828370 7fc5b9b29700 5 -- op tracker -- seq: 47362, time: 2014-11-08 18:15:36.828370, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1059> 2014-11-08 18:15:36.834145 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d39cfc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb02c0).accept sd=135 192.168.1.16:39186/0 -1058> 2014-11-08 18:15:36.843365 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302d580 con 0x23bb02c0 -1057> 2014-11-08 18:15:36.843400 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1056> 2014-11-08 18:15:36.846602 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1d06bfc0 con 0x23bb02c0 -1055> 2014-11-08 18:15:36.846635 7fc5b9b29700 5 -- op tracker -- seq: 47363, time: 2014-11-08 18:15:36.846428, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1054> 2014-11-08 18:15:36.846644 7fc5b9b29700 5 -- op tracker -- seq: 47363, time: 2014-11-08 18:15:36.846431, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1053> 2014-11-08 18:15:36.846647 7fc5b9b29700 5 -- op tracker -- seq: 47363, time: 2014-11-08 18:15:36.846516, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1052> 2014-11-08 18:15:36.846649 7fc5b9b29700 5 -- op tracker -- seq: 47363, time: 2014-11-08 18:15:36.846630, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1051> 2014-11-08 18:15:36.846670 7fc5b9b29700 5 -- op tracker -- seq: 47363, time: 2014-11-08 18:15:36.846669, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1050> 2014-11-08 18:15:36.846675 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1049> 2014-11-08 18:15:36.846681 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb02c0 -- 0x1d39cfc0 -1048> 2014-11-08 18:15:36.846717 7fc5b9b29700 5 -- op tracker -- seq: 47363, time: 2014-11-08 18:15:36.846717, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1047> 2014-11-08 18:15:36.846730 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1d0694a0 con 0x23bb02c0 -1046> 2014-11-08 18:15:36.846737 7fc5b9b29700 5 -- op tracker -- seq: 47364, time: 2014-11-08 18:15:36.846530, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1045> 2014-11-08 18:15:36.846740 7fc5b9b29700 5 -- op tracker -- seq: 47364, time: 2014-11-08 18:15:36.846531, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1044> 2014-11-08 18:15:36.846742 7fc5b9b29700 5 -- op tracker -- seq: 47364, time: 2014-11-08 18:15:36.846567, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1043> 2014-11-08 18:15:36.846744 7fc5b9b29700 5 -- op tracker -- seq: 47364, time: 2014-11-08 18:15:36.846736, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1042> 2014-11-08 18:15:36.846747 7fc5b9b29700 5 -- op tracker -- seq: 47364, time: 2014-11-08 18:15:36.846747, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1041> 2014-11-08 18:15:36.846750 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1040> 2014-11-08 18:15:36.846753 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb02c0 -- pipe dne -1039> 2014-11-08 18:15:36.846756 7fc5b9b29700 5 -- op tracker -- seq: 47364, time: 2014-11-08 18:15:36.846756, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1038> 2014-11-08 18:15:36.846762 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1d069fe0 con 0x23bb02c0 -1037> 2014-11-08 18:15:36.846768 7fc5b9b29700 5 -- op tracker -- seq: 47365, time: 2014-11-08 18:15:36.846572, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1036> 2014-11-08 18:15:36.846770 7fc5b9b29700 5 -- op tracker -- seq: 47365, time: 2014-11-08 18:15:36.846572, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1035> 2014-11-08 18:15:36.846773 7fc5b9b29700 5 -- op tracker -- seq: 47365, time: 2014-11-08 18:15:36.846591, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1034> 2014-11-08 18:15:36.846775 7fc5b9b29700 5 -- op tracker -- seq: 47365, time: 2014-11-08 18:15:36.846767, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1033> 2014-11-08 18:15:36.846778 7fc5b9b29700 5 -- op tracker -- seq: 47365, time: 2014-11-08 18:15:36.846777, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1032> 2014-11-08 18:15:36.846780 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1031> 2014-11-08 18:15:36.846783 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb02c0 -- pipe dne -1030> 2014-11-08 18:15:36.846785 7fc5b9b29700 5 -- op tracker -- seq: 47365, time: 2014-11-08 18:15:36.846785, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1029> 2014-11-08 18:15:36.846828 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d39cfc0 sd=135 :6811 s=4 pgs=8246 cs=25 l=0 c=0x23bb02c0).reader couldn't read tag, (0) Success -1028> 2014-11-08 18:15:36.846847 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d39cfc0 sd=135 :6811 s=4 pgs=8246 cs=25 l=0 c=0x23bb02c0).fault (0) Success -1027> 2014-11-08 18:15:36.853518 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d39bf40 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0420).accept sd=135 192.168.1.16:39187/0 -1026> 2014-11-08 18:15:36.859085 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23028900 con 0x23bb0420 -1025> 2014-11-08 18:15:36.859136 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1024> 2014-11-08 18:15:36.859188 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21278b40 con 0x23bb0420 -1023> 2014-11-08 18:15:36.859217 7fc5b9b29700 5 -- op tracker -- seq: 47366, time: 2014-11-08 18:15:36.859156, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1022> 2014-11-08 18:15:36.859225 7fc5b9b29700 5 -- op tracker -- seq: 47366, time: 2014-11-08 18:15:36.859157, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1021> 2014-11-08 18:15:36.859227 7fc5b9b29700 5 -- op tracker -- seq: 47366, time: 2014-11-08 18:15:36.859180, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1020> 2014-11-08 18:15:36.859230 7fc5b9b29700 5 -- op tracker -- seq: 47366, time: 2014-11-08 18:15:36.859214, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1019> 2014-11-08 18:15:36.859233 7fc5b9b29700 5 -- op tracker -- seq: 47366, time: 2014-11-08 18:15:36.859233, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1018> 2014-11-08 18:15:36.859238 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1017> 2014-11-08 18:15:36.859243 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0420 -- 0x1d39bf40 -1016> 2014-11-08 18:15:36.859267 7fc5b9b29700 5 -- op tracker -- seq: 47366, time: 2014-11-08 18:15:36.859267, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1015> 2014-11-08 18:15:36.859285 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d39bf40 sd=135 :6811 s=4 pgs=8247 cs=27 l=0 c=0x23bb0420).reader couldn't read tag, (0) Success -1014> 2014-11-08 18:15:36.859316 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d39bf40 sd=135 :6811 s=4 pgs=8247 cs=27 l=0 c=0x23bb0420).fault (0) Success -1013> 2014-11-08 18:15:36.860277 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d399b80 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0580).accept sd=135 192.168.1.16:39188/0 -1012> 2014-11-08 18:15:36.861536 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302bf00 con 0x23bb0580 -1011> 2014-11-08 18:15:36.861557 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -1010> 2014-11-08 18:15:36.861742 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x212783c0 con 0x23bb0580 -1009> 2014-11-08 18:15:36.861753 7fc5b9b29700 5 -- op tracker -- seq: 47367, time: 2014-11-08 18:15:36.861681, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -1008> 2014-11-08 18:15:36.861758 7fc5b9b29700 5 -- op tracker -- seq: 47367, time: 2014-11-08 18:15:36.861683, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -1007> 2014-11-08 18:15:36.861761 7fc5b9b29700 5 -- op tracker -- seq: 47367, time: 2014-11-08 18:15:36.861732, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -1006> 2014-11-08 18:15:36.861763 7fc5b9b29700 5 -- op tracker -- seq: 47367, time: 2014-11-08 18:15:36.861751, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -1005> 2014-11-08 18:15:36.861766 7fc5b9b29700 5 -- op tracker -- seq: 47367, time: 2014-11-08 18:15:36.861766, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -1004> 2014-11-08 18:15:36.861771 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -1003> 2014-11-08 18:15:36.861775 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0580 -- 0x1d399b80 -1002> 2014-11-08 18:15:36.861798 7fc5b9b29700 5 -- op tracker -- seq: 47367, time: 2014-11-08 18:15:36.861798, event: done, op: pg_trim(4.c to 5734'8057 e5773) -1001> 2014-11-08 18:15:36.861814 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d399b80 sd=135 :6811 s=4 pgs=8248 cs=29 l=0 c=0x23bb0580).reader couldn't read tag, (0) Success -1000> 2014-11-08 18:15:36.861830 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d399b80 sd=135 :6811 s=4 pgs=8248 cs=29 l=0 c=0x23bb0580).fault (0) Success -999> 2014-11-08 18:15:36.862945 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d28f0c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb1e40).accept sd=135 192.168.1.16:39189/0 -998> 2014-11-08 18:15:36.864518 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302b600 con 0x23bb1e40 -997> 2014-11-08 18:15:36.864534 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -996> 2014-11-08 18:15:36.864556 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127a760 con 0x23bb1e40 -995> 2014-11-08 18:15:36.864565 7fc5b9b29700 5 -- op tracker -- seq: 47368, time: 2014-11-08 18:15:36.864519, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -994> 2014-11-08 18:15:36.864571 7fc5b9b29700 5 -- op tracker -- seq: 47368, time: 2014-11-08 18:15:36.864520, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -993> 2014-11-08 18:15:36.864575 7fc5b9b29700 5 -- op tracker -- seq: 47368, time: 2014-11-08 18:15:36.864549, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -992> 2014-11-08 18:15:36.864579 7fc5b9b29700 5 -- op tracker -- seq: 47368, time: 2014-11-08 18:15:36.864564, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -991> 2014-11-08 18:15:36.864585 7fc5b9b29700 5 -- op tracker -- seq: 47368, time: 2014-11-08 18:15:36.864584, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -990> 2014-11-08 18:15:36.864590 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -989> 2014-11-08 18:15:36.864595 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb1e40 -- 0x1d28f0c0 -988> 2014-11-08 18:15:36.864607 7fc5b9b29700 5 -- op tracker -- seq: 47368, time: 2014-11-08 18:15:36.864607, event: done, op: pg_trim(4.c to 5734'8057 e5773) -987> 2014-11-08 18:15:36.865255 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d289b80 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1eb4aaa0).accept sd=135 192.168.1.16:39190/0 -986> 2014-11-08 18:15:36.866811 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23029b00 con 0x1eb4aaa0 -985> 2014-11-08 18:15:36.866825 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -984> 2014-11-08 18:15:36.866830 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127a3a0 con 0x1eb4aaa0 -983> 2014-11-08 18:15:36.866836 7fc5b9b29700 5 -- op tracker -- seq: 47369, time: 2014-11-08 18:15:36.866742, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -982> 2014-11-08 18:15:36.866840 7fc5b9b29700 5 -- op tracker -- seq: 47369, time: 2014-11-08 18:15:36.866742, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -981> 2014-11-08 18:15:36.866842 7fc5b9b29700 5 -- op tracker -- seq: 47369, time: 2014-11-08 18:15:36.866761, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -980> 2014-11-08 18:15:36.866844 7fc5b9b29700 5 -- op tracker -- seq: 47369, time: 2014-11-08 18:15:36.866835, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -979> 2014-11-08 18:15:36.866847 7fc5b9b29700 5 -- op tracker -- seq: 47369, time: 2014-11-08 18:15:36.866847, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -978> 2014-11-08 18:15:36.866850 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -977> 2014-11-08 18:15:36.866854 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1eb4aaa0 -- 0x1d289b80 -976> 2014-11-08 18:15:36.866865 7fc5b9b29700 5 -- op tracker -- seq: 47369, time: 2014-11-08 18:15:36.866865, event: done, op: pg_trim(4.c to 5734'8057 e5773) -975> 2014-11-08 18:15:36.866873 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x212792c0 con 0x1eb4aaa0 -974> 2014-11-08 18:15:36.866879 7fc5b9b29700 5 -- op tracker -- seq: 47370, time: 2014-11-08 18:15:36.866766, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -973> 2014-11-08 18:15:36.866882 7fc5b9b29700 5 -- op tracker -- seq: 47370, time: 2014-11-08 18:15:36.866766, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -972> 2014-11-08 18:15:36.866884 7fc5b9b29700 5 -- op tracker -- seq: 47370, time: 2014-11-08 18:15:36.866782, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -971> 2014-11-08 18:15:36.866887 7fc5b9b29700 5 -- op tracker -- seq: 47370, time: 2014-11-08 18:15:36.866878, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -970> 2014-11-08 18:15:36.866889 7fc5b9b29700 5 -- op tracker -- seq: 47370, time: 2014-11-08 18:15:36.866889, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -969> 2014-11-08 18:15:36.866892 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -968> 2014-11-08 18:15:36.866895 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1eb4aaa0 -- pipe dne -967> 2014-11-08 18:15:36.866898 7fc5b9b29700 5 -- op tracker -- seq: 47370, time: 2014-11-08 18:15:36.866898, event: done, op: pg_trim(4.c to 5734'8057 e5773) -966> 2014-11-08 18:15:36.866933 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21279fe0 con 0x1eb4aaa0 -965> 2014-11-08 18:15:36.866939 7fc5b9b29700 5 -- op tracker -- seq: 47371, time: 2014-11-08 18:15:36.866787, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -964> 2014-11-08 18:15:36.866943 7fc5b9b29700 5 -- op tracker -- seq: 47371, time: 2014-11-08 18:15:36.866787, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -963> 2014-11-08 18:15:36.866945 7fc5b9b29700 5 -- op tracker -- seq: 47371, time: 2014-11-08 18:15:36.866805, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -962> 2014-11-08 18:15:36.866948 7fc5b9b29700 5 -- op tracker -- seq: 47371, time: 2014-11-08 18:15:36.866938, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -961> 2014-11-08 18:15:36.866951 7fc5b9b29700 5 -- op tracker -- seq: 47371, time: 2014-11-08 18:15:36.866950, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -960> 2014-11-08 18:15:36.866953 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -959> 2014-11-08 18:15:36.866956 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1eb4aaa0 -- pipe dne -958> 2014-11-08 18:15:36.866959 7fc5b9b29700 5 -- op tracker -- seq: 47371, time: 2014-11-08 18:15:36.866959, event: done, op: pg_trim(4.c to 5734'8057 e5773) -957> 2014-11-08 18:15:36.866982 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d289b80 sd=135 :6811 s=4 pgs=8250 cs=33 l=0 c=0x1eb4aaa0).reader couldn't read tag, (0) Success -956> 2014-11-08 18:15:36.866993 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1d289b80 sd=135 :6811 s=4 pgs=8250 cs=33 l=0 c=0x1eb4aaa0).fault (0) Success -955> 2014-11-08 18:15:36.867504 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1d2898c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1eb4aec0).accept sd=135 192.168.1.16:39192/0 -954> 2014-11-08 18:15:36.869258 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302b840 con 0x1eb4aec0 -953> 2014-11-08 18:15:36.869274 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -952> 2014-11-08 18:15:36.869567 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21279c20 con 0x1eb4aec0 -951> 2014-11-08 18:15:36.869578 7fc5b9b29700 5 -- op tracker -- seq: 47372, time: 2014-11-08 18:15:36.869530, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -950> 2014-11-08 18:15:36.869583 7fc5b9b29700 5 -- op tracker -- seq: 47372, time: 2014-11-08 18:15:36.869531, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -949> 2014-11-08 18:15:36.869587 7fc5b9b29700 5 -- op tracker -- seq: 47372, time: 2014-11-08 18:15:36.869559, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -948> 2014-11-08 18:15:36.869592 7fc5b9b29700 5 -- op tracker -- seq: 47372, time: 2014-11-08 18:15:36.869576, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -947> 2014-11-08 18:15:36.869597 7fc5b9b29700 5 -- op tracker -- seq: 47372, time: 2014-11-08 18:15:36.869597, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -946> 2014-11-08 18:15:36.869602 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -945> 2014-11-08 18:15:36.869607 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1eb4aec0 -- 0x1d2898c0 -944> 2014-11-08 18:15:36.869621 7fc5b9b29700 5 -- op tracker -- seq: 47372, time: 2014-11-08 18:15:36.869621, event: done, op: pg_trim(4.c to 5734'8057 e5773) -943> 2014-11-08 18:15:36.871060 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x1ebec200 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e8d2c00).accept sd=135 192.168.1.16:39193/0 -942> 2014-11-08 18:15:36.874025 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23029440 con 0x1e8d2c00 -941> 2014-11-08 18:15:36.874051 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -940> 2014-11-08 18:15:36.874099 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21278960 con 0x1e8d2c00 -939> 2014-11-08 18:15:36.874110 7fc5b9b29700 5 -- op tracker -- seq: 47373, time: 2014-11-08 18:15:36.874067, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -938> 2014-11-08 18:15:36.874117 7fc5b9b29700 5 -- op tracker -- seq: 47373, time: 2014-11-08 18:15:36.874067, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -937> 2014-11-08 18:15:36.874120 7fc5b9b29700 5 -- op tracker -- seq: 47373, time: 2014-11-08 18:15:36.874090, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -936> 2014-11-08 18:15:36.874122 7fc5b9b29700 5 -- op tracker -- seq: 47373, time: 2014-11-08 18:15:36.874108, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -935> 2014-11-08 18:15:36.874125 7fc5b9b29700 5 -- op tracker -- seq: 47373, time: 2014-11-08 18:15:36.874125, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -934> 2014-11-08 18:15:36.874130 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -933> 2014-11-08 18:15:36.874134 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e8d2c00 -- 0x1ebec200 -932> 2014-11-08 18:15:36.874213 7fc5b9b29700 5 -- op tracker -- seq: 47373, time: 2014-11-08 18:15:36.874213, event: done, op: pg_trim(4.c to 5734'8057 e5773) -931> 2014-11-08 18:15:36.874226 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127cce0 con 0x1e8d2c00 -930> 2014-11-08 18:15:36.874232 7fc5b9b29700 5 -- op tracker -- seq: 47374, time: 2014-11-08 18:15:36.874150, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -929> 2014-11-08 18:15:36.874236 7fc5b9b29700 5 -- op tracker -- seq: 47374, time: 2014-11-08 18:15:36.874150, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -928> 2014-11-08 18:15:36.874238 7fc5b9b29700 5 -- op tracker -- seq: 47374, time: 2014-11-08 18:15:36.874171, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -927> 2014-11-08 18:15:36.874240 7fc5b9b29700 5 -- op tracker -- seq: 47374, time: 2014-11-08 18:15:36.874231, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -926> 2014-11-08 18:15:36.874243 7fc5b9b29700 5 -- op tracker -- seq: 47374, time: 2014-11-08 18:15:36.874243, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -925> 2014-11-08 18:15:36.874246 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -924> 2014-11-08 18:15:36.874249 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e8d2c00 -- pipe dne -923> 2014-11-08 18:15:36.874252 7fc5b9b29700 5 -- op tracker -- seq: 47374, time: 2014-11-08 18:15:36.874252, event: done, op: pg_trim(4.c to 5734'8057 e5773) -922> 2014-11-08 18:15:36.874333 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127bfc0 con 0x1e8d2c00 -921> 2014-11-08 18:15:36.874363 7fc5b9b29700 5 -- op tracker -- seq: 47375, time: 2014-11-08 18:15:36.874175, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -920> 2014-11-08 18:15:36.874367 7fc5b9b29700 5 -- op tracker -- seq: 47375, time: 2014-11-08 18:15:36.874176, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -919> 2014-11-08 18:15:36.874369 7fc5b9b29700 5 -- op tracker -- seq: 47375, time: 2014-11-08 18:15:36.874193, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -918> 2014-11-08 18:15:36.874371 7fc5b9b29700 5 -- op tracker -- seq: 47375, time: 2014-11-08 18:15:36.874338, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -917> 2014-11-08 18:15:36.874374 7fc5b9b29700 5 -- op tracker -- seq: 47375, time: 2014-11-08 18:15:36.874374, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -916> 2014-11-08 18:15:36.874377 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -915> 2014-11-08 18:15:36.874380 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e8d2c00 -- pipe dne -914> 2014-11-08 18:15:36.874402 7fc5b9b29700 5 -- op tracker -- seq: 47375, time: 2014-11-08 18:15:36.874402, event: done, op: pg_trim(4.c to 5734'8057 e5773) -913> 2014-11-08 18:15:36.874422 7fc5a27c0700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1ebec200 sd=135 :6811 s=4 pgs=8253 cs=37 l=0 c=0x1e8d2c00).do_sendmsg error (32) Broken pipe -912> 2014-11-08 18:15:36.874454 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1ebec200 sd=135 :6811 s=4 pgs=8253 cs=37 l=0 c=0x1e8d2c00).writer couldn't write ack, (32) Broken pipe -911> 2014-11-08 18:15:36.874468 7fc5a27c0700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1ebec200 sd=135 :6811 s=4 pgs=8253 cs=37 l=0 c=0x1e8d2c00).fault (32) Broken pipe -910> 2014-11-08 18:15:36.874660 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1ebec200 sd=135 :6811 s=4 pgs=8253 cs=37 l=0 c=0x1e8d2c00).reader couldn't read tag, (0) Success -909> 2014-11-08 18:15:36.874675 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x1ebec200 sd=135 :6811 s=4 pgs=8253 cs=37 l=0 c=0x1e8d2c00).fault (0) Success -908> 2014-11-08 18:15:36.876844 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22449600 sd=142 :6811 s=0 pgs=0 cs=0 l=0 c=0x22d1ce60).accept sd=142 192.168.1.16:39194/0 -907> 2014-11-08 18:15:36.880141 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302e0c0 con 0x22d1ce60 -906> 2014-11-08 18:15:36.880170 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -905> 2014-11-08 18:15:36.880389 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127cb00 con 0x22d1ce60 -904> 2014-11-08 18:15:36.880404 7fc5b9b29700 5 -- op tracker -- seq: 47376, time: 2014-11-08 18:15:36.880356, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -903> 2014-11-08 18:15:36.880411 7fc5b9b29700 5 -- op tracker -- seq: 47376, time: 2014-11-08 18:15:36.880357, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -902> 2014-11-08 18:15:36.880414 7fc5b9b29700 5 -- op tracker -- seq: 47376, time: 2014-11-08 18:15:36.880381, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -901> 2014-11-08 18:15:36.880416 7fc5b9b29700 5 -- op tracker -- seq: 47376, time: 2014-11-08 18:15:36.880402, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -900> 2014-11-08 18:15:36.880419 7fc5b9b29700 5 -- op tracker -- seq: 47376, time: 2014-11-08 18:15:36.880419, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -899> 2014-11-08 18:15:36.880424 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -898> 2014-11-08 18:15:36.880429 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d1ce60 -- 0x22449600 -897> 2014-11-08 18:15:36.880487 7fc5b9b29700 5 -- op tracker -- seq: 47376, time: 2014-11-08 18:15:36.880487, event: done, op: pg_trim(4.c to 5734'8057 e5773) -896> 2014-11-08 18:15:36.880522 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=142 :6811 s=4 pgs=8254 cs=39 l=0 c=0x22d1ce60).reader couldn't read tag, (0) Success -895> 2014-11-08 18:15:36.880540 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=142 :6811 s=4 pgs=8254 cs=39 l=0 c=0x22d1ce60).fault (0) Success -894> 2014-11-08 18:15:36.882744 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x224498c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x22d18c60).accept sd=135 192.168.1.16:39195/0 -893> 2014-11-08 18:15:36.886170 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302ca40 con 0x22d18c60 -892> 2014-11-08 18:15:36.886192 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -891> 2014-11-08 18:15:36.887418 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127dbe0 con 0x22d18c60 -890> 2014-11-08 18:15:36.887437 7fc5b9b29700 5 -- op tracker -- seq: 47377, time: 2014-11-08 18:15:36.887356, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -889> 2014-11-08 18:15:36.887444 7fc5b9b29700 5 -- op tracker -- seq: 47377, time: 2014-11-08 18:15:36.887358, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -888> 2014-11-08 18:15:36.887446 7fc5b9b29700 5 -- op tracker -- seq: 47377, time: 2014-11-08 18:15:36.887403, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -887> 2014-11-08 18:15:36.887449 7fc5b9b29700 5 -- op tracker -- seq: 47377, time: 2014-11-08 18:15:36.887434, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -886> 2014-11-08 18:15:36.887452 7fc5b9b29700 5 -- op tracker -- seq: 47377, time: 2014-11-08 18:15:36.887452, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -885> 2014-11-08 18:15:36.887458 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -884> 2014-11-08 18:15:36.887462 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d18c60 -- 0x224498c0 -883> 2014-11-08 18:15:36.887489 7fc5b9b29700 5 -- op tracker -- seq: 47377, time: 2014-11-08 18:15:36.887489, event: done, op: pg_trim(4.c to 5734'8057 e5773) -882> 2014-11-08 18:15:36.887506 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x224498c0 sd=135 :6811 s=4 pgs=8255 cs=41 l=0 c=0x22d18c60).reader couldn't read tag, (0) Success -881> 2014-11-08 18:15:36.887526 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x224498c0 sd=135 :6811 s=4 pgs=8255 cs=41 l=0 c=0x22d18c60).fault (0) Success -880> 2014-11-08 18:15:36.889963 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22449b80 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x22d18b00).accept sd=135 192.168.1.16:39196/0 -879> 2014-11-08 18:15:36.894220 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302d100 con 0x22d18b00 -878> 2014-11-08 18:15:36.894247 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -877> 2014-11-08 18:15:36.894839 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127d0a0 con 0x22d18b00 -876> 2014-11-08 18:15:36.894852 7fc5b9b29700 5 -- op tracker -- seq: 47378, time: 2014-11-08 18:15:36.894788, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -875> 2014-11-08 18:15:36.894862 7fc5b9b29700 5 -- op tracker -- seq: 47378, time: 2014-11-08 18:15:36.894789, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -874> 2014-11-08 18:15:36.894865 7fc5b9b29700 5 -- op tracker -- seq: 47378, time: 2014-11-08 18:15:36.894828, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -873> 2014-11-08 18:15:36.894867 7fc5b9b29700 5 -- op tracker -- seq: 47378, time: 2014-11-08 18:15:36.894850, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -872> 2014-11-08 18:15:36.894870 7fc5b9b29700 5 -- op tracker -- seq: 47378, time: 2014-11-08 18:15:36.894870, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -871> 2014-11-08 18:15:36.894875 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -870> 2014-11-08 18:15:36.894882 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d18b00 -- 0x22449b80 -869> 2014-11-08 18:15:36.894965 7fc5b9b29700 5 -- op tracker -- seq: 47378, time: 2014-11-08 18:15:36.894965, event: done, op: pg_trim(4.c to 5734'8057 e5773) -868> 2014-11-08 18:15:36.894980 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127e360 con 0x22d18b00 -867> 2014-11-08 18:15:36.894987 7fc5b9b29700 5 -- op tracker -- seq: 47379, time: 2014-11-08 18:15:36.894894, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -866> 2014-11-08 18:15:36.894990 7fc5b9b29700 5 -- op tracker -- seq: 47379, time: 2014-11-08 18:15:36.894895, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -865> 2014-11-08 18:15:36.894993 7fc5b9b29700 5 -- op tracker -- seq: 47379, time: 2014-11-08 18:15:36.894917, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -864> 2014-11-08 18:15:36.894996 7fc5b9b29700 5 -- op tracker -- seq: 47379, time: 2014-11-08 18:15:36.894986, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -863> 2014-11-08 18:15:36.894998 7fc5b9b29700 5 -- op tracker -- seq: 47379, time: 2014-11-08 18:15:36.894998, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -862> 2014-11-08 18:15:36.895002 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -861> 2014-11-08 18:15:36.895005 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d18b00 -- pipe dne -860> 2014-11-08 18:15:36.894992 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449b80 sd=135 :6811 s=4 pgs=8256 cs=43 l=0 c=0x22d18b00).reader couldn't read tag, (0) Success -859> 2014-11-08 18:15:36.895008 7fc5b9b29700 5 -- op tracker -- seq: 47379, time: 2014-11-08 18:15:36.895008, event: done, op: pg_trim(4.c to 5734'8057 e5773) -858> 2014-11-08 18:15:36.895009 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449b80 sd=135 :6811 s=4 pgs=8256 cs=43 l=0 c=0x22d18b00).fault (0) Success -857> 2014-11-08 18:15:36.895013 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127e720 con 0x22d18b00 -856> 2014-11-08 18:15:36.895019 7fc5b9b29700 5 -- op tracker -- seq: 47380, time: 2014-11-08 18:15:36.894922, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -855> 2014-11-08 18:15:36.895022 7fc5b9b29700 5 -- op tracker -- seq: 47380, time: 2014-11-08 18:15:36.894922, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -854> 2014-11-08 18:15:36.895025 7fc5b9b29700 5 -- op tracker -- seq: 47380, time: 2014-11-08 18:15:36.894939, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -853> 2014-11-08 18:15:36.895027 7fc5b9b29700 5 -- op tracker -- seq: 47380, time: 2014-11-08 18:15:36.895018, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -852> 2014-11-08 18:15:36.895030 7fc5b9b29700 5 -- op tracker -- seq: 47380, time: 2014-11-08 18:15:36.895029, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -851> 2014-11-08 18:15:36.895032 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -850> 2014-11-08 18:15:36.895035 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d18b00 -- pipe dne -849> 2014-11-08 18:15:36.895038 7fc5b9b29700 5 -- op tracker -- seq: 47380, time: 2014-11-08 18:15:36.895037, event: done, op: pg_trim(4.c to 5734'8057 e5773) -848> 2014-11-08 18:15:36.899003 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22448dc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x22d1cfc0).accept sd=135 192.168.1.16:39197/0 -847> 2014-11-08 18:15:36.904333 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302e300 con 0x22d1cfc0 -846> 2014-11-08 18:15:36.904360 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -845> 2014-11-08 18:15:36.906167 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127c740 con 0x22d1cfc0 -844> 2014-11-08 18:15:36.906180 7fc5b9b29700 5 -- op tracker -- seq: 47381, time: 2014-11-08 18:15:36.906124, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -843> 2014-11-08 18:15:36.906189 7fc5b9b29700 5 -- op tracker -- seq: 47381, time: 2014-11-08 18:15:36.906125, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -842> 2014-11-08 18:15:36.906193 7fc5b9b29700 5 -- op tracker -- seq: 47381, time: 2014-11-08 18:15:36.906156, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -841> 2014-11-08 18:15:36.906198 7fc5b9b29700 5 -- op tracker -- seq: 47381, time: 2014-11-08 18:15:36.906178, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -840> 2014-11-08 18:15:36.906202 7fc5b9b29700 5 -- op tracker -- seq: 47381, time: 2014-11-08 18:15:36.906202, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -839> 2014-11-08 18:15:36.906209 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -838> 2014-11-08 18:15:36.906215 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d1cfc0 -- 0x22448dc0 -837> 2014-11-08 18:15:36.906241 7fc5b9b29700 5 -- op tracker -- seq: 47381, time: 2014-11-08 18:15:36.906241, event: done, op: pg_trim(4.c to 5734'8057 e5773) -836> 2014-11-08 18:15:36.906239 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448dc0 sd=135 :6811 s=4 pgs=8257 cs=45 l=0 c=0x22d1cfc0).writer couldn't write ack, (4) Interrupted system call -835> 2014-11-08 18:15:36.906269 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448dc0 sd=135 :6811 s=4 pgs=8257 cs=45 l=0 c=0x22d1cfc0).fault (4) Interrupted system call -834> 2014-11-08 18:15:36.906287 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448dc0 sd=135 :6811 s=4 pgs=8257 cs=45 l=0 c=0x22d1cfc0).fault (0) Success -833> 2014-11-08 18:15:36.910456 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22448000 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x22d191e0).accept sd=135 192.168.1.16:39198/0 -832> 2014-11-08 18:15:36.917115 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302af40 con 0x22d191e0 -831> 2014-11-08 18:15:36.917146 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -830> 2014-11-08 18:15:36.919488 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x212794a0 con 0x22d191e0 -829> 2014-11-08 18:15:36.919507 7fc5b9b29700 5 -- op tracker -- seq: 47382, time: 2014-11-08 18:15:36.919412, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -828> 2014-11-08 18:15:36.919514 7fc5b9b29700 5 -- op tracker -- seq: 47382, time: 2014-11-08 18:15:36.919415, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -827> 2014-11-08 18:15:36.919517 7fc5b9b29700 5 -- op tracker -- seq: 47382, time: 2014-11-08 18:15:36.919474, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -826> 2014-11-08 18:15:36.919519 7fc5b9b29700 5 -- op tracker -- seq: 47382, time: 2014-11-08 18:15:36.919503, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -825> 2014-11-08 18:15:36.919522 7fc5b9b29700 5 -- op tracker -- seq: 47382, time: 2014-11-08 18:15:36.919522, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -824> 2014-11-08 18:15:36.919527 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -823> 2014-11-08 18:15:36.919532 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d191e0 -- 0x22448000 -822> 2014-11-08 18:15:36.919630 7fc5b9b29700 5 -- op tracker -- seq: 47382, time: 2014-11-08 18:15:36.919630, event: done, op: pg_trim(4.c to 5734'8057 e5773) -821> 2014-11-08 18:15:36.919674 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127a940 con 0x22d191e0 -820> 2014-11-08 18:15:36.919681 7fc5b9b29700 5 -- op tracker -- seq: 47383, time: 2014-11-08 18:15:36.919548, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -819> 2014-11-08 18:15:36.919698 7fc5b9b29700 5 -- op tracker -- seq: 47383, time: 2014-11-08 18:15:36.919548, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -818> 2014-11-08 18:15:36.919701 7fc5b9b29700 5 -- op tracker -- seq: 47383, time: 2014-11-08 18:15:36.919570, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -817> 2014-11-08 18:15:36.919704 7fc5b9b29700 5 -- op tracker -- seq: 47383, time: 2014-11-08 18:15:36.919680, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -816> 2014-11-08 18:15:36.919707 7fc5b9b29700 5 -- op tracker -- seq: 47383, time: 2014-11-08 18:15:36.919707, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -815> 2014-11-08 18:15:36.919710 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -814> 2014-11-08 18:15:36.919713 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d191e0 -- pipe dne -813> 2014-11-08 18:15:36.919716 7fc5b9b29700 5 -- op tracker -- seq: 47383, time: 2014-11-08 18:15:36.919716, event: done, op: pg_trim(4.c to 5734'8057 e5773) -812> 2014-11-08 18:15:36.919722 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127ab20 con 0x22d191e0 -811> 2014-11-08 18:15:36.919728 7fc5b9b29700 5 -- op tracker -- seq: 47384, time: 2014-11-08 18:15:36.919575, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -810> 2014-11-08 18:15:36.919731 7fc5b9b29700 5 -- op tracker -- seq: 47384, time: 2014-11-08 18:15:36.919575, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -809> 2014-11-08 18:15:36.919733 7fc5b9b29700 5 -- op tracker -- seq: 47384, time: 2014-11-08 18:15:36.919593, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -808> 2014-11-08 18:15:36.919735 7fc5b9b29700 5 -- op tracker -- seq: 47384, time: 2014-11-08 18:15:36.919727, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -807> 2014-11-08 18:15:36.919738 7fc5b9b29700 5 -- op tracker -- seq: 47384, time: 2014-11-08 18:15:36.919738, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -806> 2014-11-08 18:15:36.919741 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -805> 2014-11-08 18:15:36.919744 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x22d191e0 -- pipe dne -804> 2014-11-08 18:15:36.919746 7fc5b9b29700 5 -- op tracker -- seq: 47384, time: 2014-11-08 18:15:36.919746, event: done, op: pg_trim(4.c to 5734'8057 e5773) -803> 2014-11-08 18:15:36.919766 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8258 cs=47 l=0 c=0x22d191e0).do_sendmsg error (32) Broken pipe -802> 2014-11-08 18:15:36.919800 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8258 cs=47 l=0 c=0x22d191e0).writer couldn't write ack, (32) Broken pipe -801> 2014-11-08 18:15:36.919814 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8258 cs=47 l=0 c=0x22d191e0).fault (32) Broken pipe -800> 2014-11-08 18:15:36.919865 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8258 cs=47 l=0 c=0x22d191e0).reader couldn't read tag, (0) Success -799> 2014-11-08 18:15:36.919879 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8258 cs=47 l=0 c=0x22d191e0).fault (0) Success -798> 2014-11-08 18:15:36.924932 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x2508d280 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1ccfc4c0).accept sd=135 192.168.1.16:39199/0 -797> 2014-11-08 18:15:36.933345 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302ba80 con 0x1ccfc4c0 -796> 2014-11-08 18:15:36.933374 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -795> 2014-11-08 18:15:36.935772 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127aee0 con 0x1ccfc4c0 -794> 2014-11-08 18:15:36.935802 7fc5b9b29700 5 -- op tracker -- seq: 47385, time: 2014-11-08 18:15:36.935714, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -793> 2014-11-08 18:15:36.935809 7fc5b9b29700 5 -- op tracker -- seq: 47385, time: 2014-11-08 18:15:36.935716, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -792> 2014-11-08 18:15:36.935812 7fc5b9b29700 5 -- op tracker -- seq: 47385, time: 2014-11-08 18:15:36.935762, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -791> 2014-11-08 18:15:36.935814 7fc5b9b29700 5 -- op tracker -- seq: 47385, time: 2014-11-08 18:15:36.935799, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -790> 2014-11-08 18:15:36.935817 7fc5b9b29700 5 -- op tracker -- seq: 47385, time: 2014-11-08 18:15:36.935817, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -789> 2014-11-08 18:15:36.935822 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -788> 2014-11-08 18:15:36.935827 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1ccfc4c0 -- 0x2508d280 -787> 2014-11-08 18:15:36.935842 7fc5b9b29700 5 -- op tracker -- seq: 47385, time: 2014-11-08 18:15:36.935841, event: done, op: pg_trim(4.c to 5734'8057 e5773) -786> 2014-11-08 18:15:36.935840 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x2508d280 sd=135 :6811 s=4 pgs=8259 cs=49 l=0 c=0x1ccfc4c0).reader couldn't read tag, (0) Success -785> 2014-11-08 18:15:36.935853 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21279860 con 0x1ccfc4c0 -784> 2014-11-08 18:15:36.935856 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x2508d280 sd=135 :6811 s=4 pgs=8259 cs=49 l=0 c=0x1ccfc4c0).fault (0) Success -783> 2014-11-08 18:15:36.935860 7fc5b9b29700 5 -- op tracker -- seq: 47386, time: 2014-11-08 18:15:36.935773, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -782> 2014-11-08 18:15:36.935863 7fc5b9b29700 5 -- op tracker -- seq: 47386, time: 2014-11-08 18:15:36.935774, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -781> 2014-11-08 18:15:36.935865 7fc5b9b29700 5 -- op tracker -- seq: 47386, time: 2014-11-08 18:15:36.935793, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -780> 2014-11-08 18:15:36.935868 7fc5b9b29700 5 -- op tracker -- seq: 47386, time: 2014-11-08 18:15:36.935859, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -779> 2014-11-08 18:15:36.935870 7fc5b9b29700 5 -- op tracker -- seq: 47386, time: 2014-11-08 18:15:36.935870, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -778> 2014-11-08 18:15:36.935873 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -777> 2014-11-08 18:15:36.935877 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1ccfc4c0 -- pipe dne -776> 2014-11-08 18:15:36.935879 7fc5b9b29700 5 -- op tracker -- seq: 47386, time: 2014-11-08 18:15:36.935879, event: done, op: pg_trim(4.c to 5734'8057 e5773) -775> 2014-11-08 18:15:36.935888 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127ad00 con 0x1ccfc4c0 -774> 2014-11-08 18:15:36.935897 7fc5b9b29700 5 -- op tracker -- seq: 47387, time: 2014-11-08 18:15:36.935797, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -773> 2014-11-08 18:15:36.935900 7fc5b9b29700 5 -- op tracker -- seq: 47387, time: 2014-11-08 18:15:36.935798, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -772> 2014-11-08 18:15:36.935902 7fc5b9b29700 5 -- op tracker -- seq: 47387, time: 2014-11-08 18:15:36.935814, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -771> 2014-11-08 18:15:36.935905 7fc5b9b29700 5 -- op tracker -- seq: 47387, time: 2014-11-08 18:15:36.935893, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -770> 2014-11-08 18:15:36.935907 7fc5b9b29700 5 -- op tracker -- seq: 47387, time: 2014-11-08 18:15:36.935907, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -769> 2014-11-08 18:15:36.935910 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -768> 2014-11-08 18:15:36.935913 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1ccfc4c0 -- pipe dne -767> 2014-11-08 18:15:36.935916 7fc5b9b29700 5 -- op tracker -- seq: 47387, time: 2014-11-08 18:15:36.935915, event: done, op: pg_trim(4.c to 5734'8057 e5773) -766> 2014-11-08 18:15:36.941831 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23bf8840 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1cec6460).accept sd=135 192.168.1.16:39200/0 -765> 2014-11-08 18:15:36.950810 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23028d80 con 0x1cec6460 -764> 2014-11-08 18:15:36.950835 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -763> 2014-11-08 18:15:36.953835 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21279a40 con 0x1cec6460 -762> 2014-11-08 18:15:36.953851 7fc5b9b29700 5 -- op tracker -- seq: 47388, time: 2014-11-08 18:15:36.953719, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -761> 2014-11-08 18:15:36.953867 7fc5b9b29700 5 -- op tracker -- seq: 47388, time: 2014-11-08 18:15:36.953721, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -760> 2014-11-08 18:15:36.953870 7fc5b9b29700 5 -- op tracker -- seq: 47388, time: 2014-11-08 18:15:36.953755, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -759> 2014-11-08 18:15:36.953873 7fc5b9b29700 5 -- op tracker -- seq: 47388, time: 2014-11-08 18:15:36.953848, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -758> 2014-11-08 18:15:36.953876 7fc5b9b29700 5 -- op tracker -- seq: 47388, time: 2014-11-08 18:15:36.953876, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -757> 2014-11-08 18:15:36.953881 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -756> 2014-11-08 18:15:36.953885 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1cec6460 -- 0x23bf8840 -755> 2014-11-08 18:15:36.953901 7fc5b9b29700 5 -- op tracker -- seq: 47388, time: 2014-11-08 18:15:36.953901, event: done, op: pg_trim(4.c to 5734'8057 e5773) -754> 2014-11-08 18:15:36.953912 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127a1c0 con 0x1cec6460 -753> 2014-11-08 18:15:36.953918 7fc5b9b29700 5 -- op tracker -- seq: 47389, time: 2014-11-08 18:15:36.953766, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -752> 2014-11-08 18:15:36.953921 7fc5b9b29700 5 -- op tracker -- seq: 47389, time: 2014-11-08 18:15:36.953767, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -751> 2014-11-08 18:15:36.953923 7fc5b9b29700 5 -- op tracker -- seq: 47389, time: 2014-11-08 18:15:36.953801, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -750> 2014-11-08 18:15:36.953926 7fc5b9b29700 5 -- op tracker -- seq: 47389, time: 2014-11-08 18:15:36.953917, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -749> 2014-11-08 18:15:36.953929 7fc5b9b29700 5 -- op tracker -- seq: 47389, time: 2014-11-08 18:15:36.953929, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -748> 2014-11-08 18:15:36.953932 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -747> 2014-11-08 18:15:36.953935 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1cec6460 -- pipe dne -746> 2014-11-08 18:15:36.953937 7fc5b9b29700 5 -- op tracker -- seq: 47389, time: 2014-11-08 18:15:36.953937, event: done, op: pg_trim(4.c to 5734'8057 e5773) -745> 2014-11-08 18:15:36.953947 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x21279680 con 0x1cec6460 -744> 2014-11-08 18:15:36.953953 7fc5b9b29700 5 -- op tracker -- seq: 47390, time: 2014-11-08 18:15:36.953808, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -743> 2014-11-08 18:15:36.953956 7fc5b9b29700 5 -- op tracker -- seq: 47390, time: 2014-11-08 18:15:36.953808, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -742> 2014-11-08 18:15:36.953958 7fc5b9b29700 5 -- op tracker -- seq: 47390, time: 2014-11-08 18:15:36.953826, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -741> 2014-11-08 18:15:36.953960 7fc5b9b29700 5 -- op tracker -- seq: 47390, time: 2014-11-08 18:15:36.953952, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -740> 2014-11-08 18:15:36.953963 7fc5b9b29700 5 -- op tracker -- seq: 47390, time: 2014-11-08 18:15:36.953963, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -739> 2014-11-08 18:15:36.953966 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -738> 2014-11-08 18:15:36.953969 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1cec6460 -- pipe dne -737> 2014-11-08 18:15:36.953971 7fc5b9b29700 5 -- op tracker -- seq: 47390, time: 2014-11-08 18:15:36.953971, event: done, op: pg_trim(4.c to 5734'8057 e5773) -736> 2014-11-08 18:15:36.953979 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bf8840 sd=135 :6811 s=4 pgs=8260 cs=51 l=0 c=0x1cec6460).reader couldn't read tag, (0) Success -735> 2014-11-08 18:15:36.953992 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bf8840 sd=135 :6811 s=4 pgs=8260 cs=51 l=0 c=0x1cec6460).fault (0) Success -734> 2014-11-08 18:15:36.959703 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23bfc200 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1cec2680).accept sd=135 192.168.1.16:39201/0 -733> 2014-11-08 18:15:36.967168 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302d340 con 0x1cec2680 -732> 2014-11-08 18:15:36.967210 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -731> 2014-11-08 18:15:36.967262 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2127a580 con 0x1cec2680 -730> 2014-11-08 18:15:36.967273 7fc5b9b29700 5 -- op tracker -- seq: 47391, time: 2014-11-08 18:15:36.967229, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -729> 2014-11-08 18:15:36.967281 7fc5b9b29700 5 -- op tracker -- seq: 47391, time: 2014-11-08 18:15:36.967229, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -728> 2014-11-08 18:15:36.967297 7fc5b9b29700 5 -- op tracker -- seq: 47391, time: 2014-11-08 18:15:36.967253, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -727> 2014-11-08 18:15:36.967301 7fc5b9b29700 5 -- op tracker -- seq: 47391, time: 2014-11-08 18:15:36.967271, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -726> 2014-11-08 18:15:36.967306 7fc5b9b29700 5 -- op tracker -- seq: 47391, time: 2014-11-08 18:15:36.967306, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -725> 2014-11-08 18:15:36.967313 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -724> 2014-11-08 18:15:36.967319 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1cec2680 -- 0x23bfc200 -723> 2014-11-08 18:15:36.967401 7fc5b9b29700 5 -- op tracker -- seq: 47391, time: 2014-11-08 18:15:36.967401, event: done, op: pg_trim(4.c to 5734'8057 e5773) -722> 2014-11-08 18:15:36.967414 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db18960 con 0x1cec2680 -721> 2014-11-08 18:15:36.967423 7fc5b9b29700 5 -- op tracker -- seq: 47392, time: 2014-11-08 18:15:36.967329, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -720> 2014-11-08 18:15:36.967428 7fc5b9b29700 5 -- op tracker -- seq: 47392, time: 2014-11-08 18:15:36.967329, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -719> 2014-11-08 18:15:36.967432 7fc5b9b29700 5 -- op tracker -- seq: 47392, time: 2014-11-08 18:15:36.967351, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -718> 2014-11-08 18:15:36.967435 7fc5b9b29700 5 -- op tracker -- seq: 47392, time: 2014-11-08 18:15:36.967422, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -717> 2014-11-08 18:15:36.967438 7fc5b9b29700 5 -- op tracker -- seq: 47392, time: 2014-11-08 18:15:36.967438, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -716> 2014-11-08 18:15:36.967443 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -715> 2014-11-08 18:15:36.967448 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1cec2680 -- pipe dne -714> 2014-11-08 18:15:36.967451 7fc5b9b29700 5 -- op tracker -- seq: 47392, time: 2014-11-08 18:15:36.967451, event: done, op: pg_trim(4.c to 5734'8057 e5773) -713> 2014-11-08 18:15:36.967458 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1d460 con 0x1cec2680 -712> 2014-11-08 18:15:36.967466 7fc5b9b29700 5 -- op tracker -- seq: 47393, time: 2014-11-08 18:15:36.967355, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -711> 2014-11-08 18:15:36.967470 7fc5b9b29700 5 -- op tracker -- seq: 47393, time: 2014-11-08 18:15:36.967356, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -710> 2014-11-08 18:15:36.967474 7fc5b9b29700 5 -- op tracker -- seq: 47393, time: 2014-11-08 18:15:36.967373, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -709> 2014-11-08 18:15:36.967478 7fc5b9b29700 5 -- op tracker -- seq: 47393, time: 2014-11-08 18:15:36.967465, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -708> 2014-11-08 18:15:36.967482 7fc5b9b29700 5 -- op tracker -- seq: 47393, time: 2014-11-08 18:15:36.967482, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -707> 2014-11-08 18:15:36.967485 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -706> 2014-11-08 18:15:36.967488 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1cec2680 -- pipe dne -705> 2014-11-08 18:15:36.967491 7fc5b9b29700 5 -- op tracker -- seq: 47393, time: 2014-11-08 18:15:36.967491, event: done, op: pg_trim(4.c to 5734'8057 e5773) -704> 2014-11-08 18:15:36.967500 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bfc200 sd=135 :6811 s=4 pgs=8261 cs=53 l=0 c=0x1cec2680).do_sendmsg error (32) Broken pipe -703> 2014-11-08 18:15:36.967523 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bfc200 sd=135 :6811 s=4 pgs=8261 cs=53 l=0 c=0x1cec2680).writer couldn't write ack, (32) Broken pipe -702> 2014-11-08 18:15:36.967533 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bfc200 sd=135 :6811 s=4 pgs=8261 cs=53 l=0 c=0x1cec2680).fault (32) Broken pipe -701> 2014-11-08 18:15:36.967610 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bfc200 sd=135 :6811 s=4 pgs=8261 cs=53 l=0 c=0x1cec2680).reader couldn't read tag, (0) Success -700> 2014-11-08 18:15:36.967621 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23bfc200 sd=135 :6811 s=4 pgs=8261 cs=53 l=0 c=0x1cec2680).fault (0) Success -699> 2014-11-08 18:15:36.968026 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x2244b9c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1d232520).accept sd=135 192.168.1.16:39202/0 -698> 2014-11-08 18:15:36.969067 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302da00 con 0x1d232520 -697> 2014-11-08 18:15:36.969083 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -696> 2014-11-08 18:15:36.969249 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1a760 con 0x1d232520 -695> 2014-11-08 18:15:36.969271 7fc5b9b29700 5 -- op tracker -- seq: 47394, time: 2014-11-08 18:15:36.969217, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -694> 2014-11-08 18:15:36.969276 7fc5b9b29700 5 -- op tracker -- seq: 47394, time: 2014-11-08 18:15:36.969218, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -693> 2014-11-08 18:15:36.969278 7fc5b9b29700 5 -- op tracker -- seq: 47394, time: 2014-11-08 18:15:36.969241, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -692> 2014-11-08 18:15:36.969281 7fc5b9b29700 5 -- op tracker -- seq: 47394, time: 2014-11-08 18:15:36.969269, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -691> 2014-11-08 18:15:36.969283 7fc5b9b29700 5 -- op tracker -- seq: 47394, time: 2014-11-08 18:15:36.969283, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -690> 2014-11-08 18:15:36.969287 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -689> 2014-11-08 18:15:36.969291 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1d232520 -- 0x2244b9c0 -688> 2014-11-08 18:15:36.969306 7fc5b9b29700 5 -- op tracker -- seq: 47394, time: 2014-11-08 18:15:36.969305, event: done, op: pg_trim(4.c to 5734'8057 e5773) -687> 2014-11-08 18:15:36.969317 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x2244b9c0 sd=135 :6811 s=4 pgs=8262 cs=55 l=0 c=0x1d232520).reader couldn't read tag, (0) Success -686> 2014-11-08 18:15:36.969326 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x2244b9c0 sd=135 :6811 s=4 pgs=8262 cs=55 l=0 c=0x1d232520).fault (0) Success -685> 2014-11-08 18:15:36.969981 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x224482c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e50ad60).accept sd=135 192.168.1.16:39203/0 -684> 2014-11-08 18:15:36.970918 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x230298c0 con 0x1e50ad60 -683> 2014-11-08 18:15:36.970936 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -682> 2014-11-08 18:15:36.971030 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1b840 con 0x1e50ad60 -681> 2014-11-08 18:15:36.971039 7fc5b9b29700 5 -- op tracker -- seq: 47395, time: 2014-11-08 18:15:36.970996, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -680> 2014-11-08 18:15:36.971044 7fc5b9b29700 5 -- op tracker -- seq: 47395, time: 2014-11-08 18:15:36.970997, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -679> 2014-11-08 18:15:36.971046 7fc5b9b29700 5 -- op tracker -- seq: 47395, time: 2014-11-08 18:15:36.971020, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -678> 2014-11-08 18:15:36.971049 7fc5b9b29700 5 -- op tracker -- seq: 47395, time: 2014-11-08 18:15:36.971037, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -677> 2014-11-08 18:15:36.971052 7fc5b9b29700 5 -- op tracker -- seq: 47395, time: 2014-11-08 18:15:36.971052, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -676> 2014-11-08 18:15:36.971056 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -675> 2014-11-08 18:15:36.971078 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ad60 -- 0x224482c0 -674> 2014-11-08 18:15:36.971094 7fc5b9b29700 5 -- op tracker -- seq: 47395, time: 2014-11-08 18:15:36.971094, event: done, op: pg_trim(4.c to 5734'8057 e5773) -673> 2014-11-08 18:15:36.971108 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x224482c0 sd=135 :6811 s=4 pgs=8263 cs=57 l=0 c=0x1e50ad60).reader couldn't read tag, (0) Success -672> 2014-11-08 18:15:36.971118 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x224482c0 sd=135 :6811 s=4 pgs=8263 cs=57 l=0 c=0x1e50ad60).fault (0) Success -671> 2014-11-08 18:15:36.971802 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22449600 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e509fa0).accept sd=135 192.168.1.16:39204/0 -670> 2014-11-08 18:15:36.972773 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302e780 con 0x1e509fa0 -669> 2014-11-08 18:15:36.972790 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -668> 2014-11-08 18:15:36.973756 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db18f00 con 0x1e509fa0 -667> 2014-11-08 18:15:36.973768 7fc5b9b29700 5 -- op tracker -- seq: 47396, time: 2014-11-08 18:15:36.973702, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -666> 2014-11-08 18:15:36.973774 7fc5b9b29700 5 -- op tracker -- seq: 47396, time: 2014-11-08 18:15:36.973703, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -665> 2014-11-08 18:15:36.973776 7fc5b9b29700 5 -- op tracker -- seq: 47396, time: 2014-11-08 18:15:36.973743, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -664> 2014-11-08 18:15:36.973779 7fc5b9b29700 5 -- op tracker -- seq: 47396, time: 2014-11-08 18:15:36.973766, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -663> 2014-11-08 18:15:36.973782 7fc5b9b29700 5 -- op tracker -- seq: 47396, time: 2014-11-08 18:15:36.973782, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -662> 2014-11-08 18:15:36.973786 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -661> 2014-11-08 18:15:36.973790 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509fa0 -- 0x22449600 -660> 2014-11-08 18:15:36.973884 7fc5b9b29700 5 -- op tracker -- seq: 47396, time: 2014-11-08 18:15:36.973884, event: done, op: pg_trim(4.c to 5734'8057 e5773) -659> 2014-11-08 18:15:36.973897 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1cb00 con 0x1e509fa0 -658> 2014-11-08 18:15:36.973903 7fc5b9b29700 5 -- op tracker -- seq: 47397, time: 2014-11-08 18:15:36.973804, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -657> 2014-11-08 18:15:36.973920 7fc5b9b29700 5 -- op tracker -- seq: 47397, time: 2014-11-08 18:15:36.973805, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -656> 2014-11-08 18:15:36.973923 7fc5b9b29700 5 -- op tracker -- seq: 47397, time: 2014-11-08 18:15:36.973843, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -655> 2014-11-08 18:15:36.973925 7fc5b9b29700 5 -- op tracker -- seq: 47397, time: 2014-11-08 18:15:36.973902, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -654> 2014-11-08 18:15:36.973928 7fc5b9b29700 5 -- op tracker -- seq: 47397, time: 2014-11-08 18:15:36.973928, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -653> 2014-11-08 18:15:36.973931 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -652> 2014-11-08 18:15:36.973935 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509fa0 -- pipe dne -651> 2014-11-08 18:15:36.973937 7fc5b9b29700 5 -- op tracker -- seq: 47397, time: 2014-11-08 18:15:36.973937, event: done, op: pg_trim(4.c to 5734'8057 e5773) -650> 2014-11-08 18:15:36.973943 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1eea0 con 0x1e509fa0 -649> 2014-11-08 18:15:36.973948 7fc5b9b29700 5 -- op tracker -- seq: 47398, time: 2014-11-08 18:15:36.973848, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -648> 2014-11-08 18:15:36.973951 7fc5b9b29700 5 -- op tracker -- seq: 47398, time: 2014-11-08 18:15:36.973848, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -647> 2014-11-08 18:15:36.973954 7fc5b9b29700 5 -- op tracker -- seq: 47398, time: 2014-11-08 18:15:36.973866, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -646> 2014-11-08 18:15:36.973956 7fc5b9b29700 5 -- op tracker -- seq: 47398, time: 2014-11-08 18:15:36.973947, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -645> 2014-11-08 18:15:36.973958 7fc5b9b29700 5 -- op tracker -- seq: 47398, time: 2014-11-08 18:15:36.973958, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -644> 2014-11-08 18:15:36.973961 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -643> 2014-11-08 18:15:36.973964 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509fa0 -- pipe dne -642> 2014-11-08 18:15:36.973967 7fc5b9b29700 5 -- op tracker -- seq: 47398, time: 2014-11-08 18:15:36.973967, event: done, op: pg_trim(4.c to 5734'8057 e5773) -641> 2014-11-08 18:15:36.973985 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=135 :6811 s=4 pgs=8264 cs=59 l=0 c=0x1e509fa0).do_sendmsg error (32) Broken pipe -640> 2014-11-08 18:15:36.974004 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=135 :6811 s=4 pgs=8264 cs=59 l=0 c=0x1e509fa0).writer couldn't write ack, (32) Broken pipe -639> 2014-11-08 18:15:36.974023 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=135 :6811 s=4 pgs=8264 cs=59 l=0 c=0x1e509fa0).fault (32) Broken pipe -638> 2014-11-08 18:15:36.974079 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=135 :6811 s=4 pgs=8264 cs=59 l=0 c=0x1e509fa0).reader couldn't read tag, (0) Success -637> 2014-11-08 18:15:36.974089 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22449600 sd=135 :6811 s=4 pgs=8264 cs=59 l=0 c=0x1e509fa0).fault (0) Success -636> 2014-11-08 18:15:36.974548 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x224498c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e509e40).accept sd=135 192.168.1.16:39205/0 -635> 2014-11-08 18:15:36.975588 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302f080 con 0x1e509e40 -634> 2014-11-08 18:15:36.975633 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -633> 2014-11-08 18:15:36.975746 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db185a0 con 0x1e509e40 -632> 2014-11-08 18:15:36.975757 7fc5b9b29700 5 -- op tracker -- seq: 47399, time: 2014-11-08 18:15:36.975712, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -631> 2014-11-08 18:15:36.975763 7fc5b9b29700 5 -- op tracker -- seq: 47399, time: 2014-11-08 18:15:36.975713, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -630> 2014-11-08 18:15:36.975766 7fc5b9b29700 5 -- op tracker -- seq: 47399, time: 2014-11-08 18:15:36.975736, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -629> 2014-11-08 18:15:36.975783 7fc5b9b29700 5 -- op tracker -- seq: 47399, time: 2014-11-08 18:15:36.975755, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -628> 2014-11-08 18:15:36.975789 7fc5b9b29700 5 -- op tracker -- seq: 47399, time: 2014-11-08 18:15:36.975789, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -627> 2014-11-08 18:15:36.975794 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -626> 2014-11-08 18:15:36.975799 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509e40 -- 0x224498c0 -625> 2014-11-08 18:15:36.975815 7fc5b9b29700 5 -- op tracker -- seq: 47399, time: 2014-11-08 18:15:36.975815, event: done, op: pg_trim(4.c to 5734'8057 e5773) -624> 2014-11-08 18:15:36.975828 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x224498c0 sd=135 :6811 s=4 pgs=8265 cs=61 l=0 c=0x1e509e40).reader couldn't read tag, (0) Success -623> 2014-11-08 18:15:36.975839 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x224498c0 sd=135 :6811 s=4 pgs=8265 cs=61 l=0 c=0x1e509e40).fault (0) Success -622> 2014-11-08 18:15:36.976539 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22449b80 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e509ce0).accept sd=135 192.168.1.16:39207/0 -621> 2014-11-08 18:15:36.978591 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302f740 con 0x1e509ce0 -620> 2014-11-08 18:15:36.978610 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -619> 2014-11-08 18:15:36.978815 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1f440 con 0x1e509ce0 -618> 2014-11-08 18:15:36.978827 7fc5b9b29700 5 -- op tracker -- seq: 47400, time: 2014-11-08 18:15:36.978778, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -617> 2014-11-08 18:15:36.978833 7fc5b9b29700 5 -- op tracker -- seq: 47400, time: 2014-11-08 18:15:36.978779, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -616> 2014-11-08 18:15:36.978837 7fc5b9b29700 5 -- op tracker -- seq: 47400, time: 2014-11-08 18:15:36.978807, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -615> 2014-11-08 18:15:36.978841 7fc5b9b29700 5 -- op tracker -- seq: 47400, time: 2014-11-08 18:15:36.978825, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -614> 2014-11-08 18:15:36.978846 7fc5b9b29700 5 -- op tracker -- seq: 47400, time: 2014-11-08 18:15:36.978846, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -613> 2014-11-08 18:15:36.978865 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -612> 2014-11-08 18:15:36.978871 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509ce0 -- 0x22449b80 -611> 2014-11-08 18:15:36.978884 7fc5b9b29700 5 -- op tracker -- seq: 47400, time: 2014-11-08 18:15:36.978884, event: done, op: pg_trim(4.c to 5734'8057 e5773) -610> 2014-11-08 18:15:36.978895 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234ba20 con 0x1e509ce0 -609> 2014-11-08 18:15:36.978903 7fc5b9b29700 5 -- op tracker -- seq: 47401, time: 2014-11-08 18:15:36.978815, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -608> 2014-11-08 18:15:36.978907 7fc5b9b29700 5 -- op tracker -- seq: 47401, time: 2014-11-08 18:15:36.978815, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -607> 2014-11-08 18:15:36.978909 7fc5b9b29700 5 -- op tracker -- seq: 47401, time: 2014-11-08 18:15:36.978853, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -606> 2014-11-08 18:15:36.978913 7fc5b9b29700 5 -- op tracker -- seq: 47401, time: 2014-11-08 18:15:36.978902, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -605> 2014-11-08 18:15:36.978918 7fc5b9b29700 5 -- op tracker -- seq: 47401, time: 2014-11-08 18:15:36.978918, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -604> 2014-11-08 18:15:36.978921 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -603> 2014-11-08 18:15:36.978924 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e509ce0 -- pipe dne -602> 2014-11-08 18:15:36.978927 7fc5b9b29700 5 -- op tracker -- seq: 47401, time: 2014-11-08 18:15:36.978927, event: done, op: pg_trim(4.c to 5734'8057 e5773) -601> 2014-11-08 18:15:36.980515 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22448dc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e50ef60).accept sd=135 192.168.1.16:39208/0 -600> 2014-11-08 18:15:36.982787 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23028b40 con 0x1e50ef60 -599> 2014-11-08 18:15:36.982804 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -598> 2014-11-08 18:15:36.983524 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x1db1f260 con 0x1e50ef60 -597> 2014-11-08 18:15:36.983536 7fc5b9b29700 5 -- op tracker -- seq: 47402, time: 2014-11-08 18:15:36.983482, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -596> 2014-11-08 18:15:36.983542 7fc5b9b29700 5 -- op tracker -- seq: 47402, time: 2014-11-08 18:15:36.983483, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -595> 2014-11-08 18:15:36.983544 7fc5b9b29700 5 -- op tracker -- seq: 47402, time: 2014-11-08 18:15:36.983509, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -594> 2014-11-08 18:15:36.983546 7fc5b9b29700 5 -- op tracker -- seq: 47402, time: 2014-11-08 18:15:36.983535, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -593> 2014-11-08 18:15:36.983549 7fc5b9b29700 5 -- op tracker -- seq: 47402, time: 2014-11-08 18:15:36.983549, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -592> 2014-11-08 18:15:36.983553 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -591> 2014-11-08 18:15:36.983557 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ef60 -- 0x22448dc0 -590> 2014-11-08 18:15:36.983640 7fc5b9b29700 5 -- op tracker -- seq: 47402, time: 2014-11-08 18:15:36.983640, event: done, op: pg_trim(4.c to 5734'8057 e5773) -589> 2014-11-08 18:15:36.983637 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448dc0 sd=135 :6811 s=4 pgs=8268 cs=65 l=0 c=0x1e50ef60).reader couldn't read tag, (0) Success -588> 2014-11-08 18:15:36.983652 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x223492c0 con 0x1e50ef60 -587> 2014-11-08 18:15:36.983670 7fc5b9b29700 5 -- op tracker -- seq: 47403, time: 2014-11-08 18:15:36.983572, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -586> 2014-11-08 18:15:36.983663 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448dc0 sd=135 :6811 s=4 pgs=8268 cs=65 l=0 c=0x1e50ef60).fault (0) Success -585> 2014-11-08 18:15:36.983680 7fc5b9b29700 5 -- op tracker -- seq: 47403, time: 2014-11-08 18:15:36.983573, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -584> 2014-11-08 18:15:36.983694 7fc5b9b29700 5 -- op tracker -- seq: 47403, time: 2014-11-08 18:15:36.983594, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -583> 2014-11-08 18:15:36.983699 7fc5b9b29700 5 -- op tracker -- seq: 47403, time: 2014-11-08 18:15:36.983668, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -582> 2014-11-08 18:15:36.983703 7fc5b9b29700 5 -- op tracker -- seq: 47403, time: 2014-11-08 18:15:36.983702, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -581> 2014-11-08 18:15:36.983706 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -580> 2014-11-08 18:15:36.983762 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ef60 -- pipe dne -579> 2014-11-08 18:15:36.983768 7fc5b9b29700 5 -- op tracker -- seq: 47403, time: 2014-11-08 18:15:36.983768, event: done, op: pg_trim(4.c to 5734'8057 e5773) -578> 2014-11-08 18:15:36.983775 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234d820 con 0x1e50ef60 -577> 2014-11-08 18:15:36.983783 7fc5b9b29700 5 -- op tracker -- seq: 47404, time: 2014-11-08 18:15:36.983599, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -576> 2014-11-08 18:15:36.983788 7fc5b9b29700 5 -- op tracker -- seq: 47404, time: 2014-11-08 18:15:36.983599, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -575> 2014-11-08 18:15:36.983794 7fc5b9b29700 5 -- op tracker -- seq: 47404, time: 2014-11-08 18:15:36.983616, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -574> 2014-11-08 18:15:36.983796 7fc5b9b29700 5 -- op tracker -- seq: 47404, time: 2014-11-08 18:15:36.983782, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -573> 2014-11-08 18:15:36.983799 7fc5b9b29700 5 -- op tracker -- seq: 47404, time: 2014-11-08 18:15:36.983799, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -572> 2014-11-08 18:15:36.983802 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -571> 2014-11-08 18:15:36.983805 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50ef60 -- pipe dne -570> 2014-11-08 18:15:36.983808 7fc5b9b29700 5 -- op tracker -- seq: 47404, time: 2014-11-08 18:15:36.983808, event: done, op: pg_trim(4.c to 5734'8057 e5773) -569> 2014-11-08 18:15:36.985446 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x22448000 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x1e50c620).accept sd=135 192.168.1.16:39209/0 -568> 2014-11-08 18:15:36.988347 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302c5c0 con 0x1e50c620 -567> 2014-11-08 18:15:36.988400 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -566> 2014-11-08 18:15:36.988833 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234b660 con 0x1e50c620 -565> 2014-11-08 18:15:36.988845 7fc5b9b29700 5 -- op tracker -- seq: 47405, time: 2014-11-08 18:15:36.988748, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -564> 2014-11-08 18:15:36.988852 7fc5b9b29700 5 -- op tracker -- seq: 47405, time: 2014-11-08 18:15:36.988749, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -563> 2014-11-08 18:15:36.988855 7fc5b9b29700 5 -- op tracker -- seq: 47405, time: 2014-11-08 18:15:36.988773, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -562> 2014-11-08 18:15:36.988859 7fc5b9b29700 5 -- op tracker -- seq: 47405, time: 2014-11-08 18:15:36.988844, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -561> 2014-11-08 18:15:36.988862 7fc5b9b29700 5 -- op tracker -- seq: 47405, time: 2014-11-08 18:15:36.988862, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -560> 2014-11-08 18:15:36.988868 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -559> 2014-11-08 18:15:36.988874 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50c620 -- 0x22448000 -558> 2014-11-08 18:15:36.988891 7fc5b9b29700 5 -- op tracker -- seq: 47405, time: 2014-11-08 18:15:36.988891, event: done, op: pg_trim(4.c to 5734'8057 e5773) -557> 2014-11-08 18:15:36.988902 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x22348780 con 0x1e50c620 -556> 2014-11-08 18:15:36.988910 7fc5b9b29700 5 -- op tracker -- seq: 47406, time: 2014-11-08 18:15:36.988782, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -555> 2014-11-08 18:15:36.988914 7fc5b9b29700 5 -- op tracker -- seq: 47406, time: 2014-11-08 18:15:36.988782, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -554> 2014-11-08 18:15:36.988917 7fc5b9b29700 5 -- op tracker -- seq: 47406, time: 2014-11-08 18:15:36.988801, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -553> 2014-11-08 18:15:36.988921 7fc5b9b29700 5 -- op tracker -- seq: 47406, time: 2014-11-08 18:15:36.988909, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -552> 2014-11-08 18:15:36.988924 7fc5b9b29700 5 -- op tracker -- seq: 47406, time: 2014-11-08 18:15:36.988924, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -551> 2014-11-08 18:15:36.988932 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -550> 2014-11-08 18:15:36.988936 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50c620 -- pipe dne -549> 2014-11-08 18:15:36.988938 7fc5b9b29700 5 -- op tracker -- seq: 47406, time: 2014-11-08 18:15:36.988938, event: done, op: pg_trim(4.c to 5734'8057 e5773) -548> 2014-11-08 18:15:36.988946 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234c740 con 0x1e50c620 -547> 2014-11-08 18:15:36.988953 7fc5b9b29700 5 -- op tracker -- seq: 47407, time: 2014-11-08 18:15:36.988808, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -546> 2014-11-08 18:15:36.988957 7fc5b9b29700 5 -- op tracker -- seq: 47407, time: 2014-11-08 18:15:36.988808, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -545> 2014-11-08 18:15:36.988961 7fc5b9b29700 5 -- op tracker -- seq: 47407, time: 2014-11-08 18:15:36.988825, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -544> 2014-11-08 18:15:36.988963 7fc5b9b29700 5 -- op tracker -- seq: 47407, time: 2014-11-08 18:15:36.988953, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -543> 2014-11-08 18:15:36.988968 7fc5b9b29700 5 -- op tracker -- seq: 47407, time: 2014-11-08 18:15:36.988968, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -542> 2014-11-08 18:15:36.988971 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -541> 2014-11-08 18:15:36.988975 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x1e50c620 -- pipe dne -540> 2014-11-08 18:15:36.988978 7fc5b9b29700 5 -- op tracker -- seq: 47407, time: 2014-11-08 18:15:36.988978, event: done, op: pg_trim(4.c to 5734'8057 e5773) -539> 2014-11-08 18:15:36.988987 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8269 cs=67 l=0 c=0x1e50c620).reader couldn't read tag, (0) Success -538> 2014-11-08 18:15:36.988999 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x22448000 sd=135 :6811 s=4 pgs=8269 cs=67 l=0 c=0x1e50c620).fault (0) Success -537> 2014-11-08 18:15:36.991213 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23724a40 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0dc0).accept sd=135 192.168.1.16:39210/0 -536> 2014-11-08 18:15:36.994770 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x1ea78240 con 0x23bb0dc0 -535> 2014-11-08 18:15:36.994789 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -534> 2014-11-08 18:15:36.995821 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234a760 con 0x23bb0dc0 -533> 2014-11-08 18:15:36.995833 7fc5b9b29700 5 -- op tracker -- seq: 47408, time: 2014-11-08 18:15:36.995710, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -532> 2014-11-08 18:15:36.995841 7fc5b9b29700 5 -- op tracker -- seq: 47408, time: 2014-11-08 18:15:36.995712, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -531> 2014-11-08 18:15:36.995844 7fc5b9b29700 5 -- op tracker -- seq: 47408, time: 2014-11-08 18:15:36.995761, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -530> 2014-11-08 18:15:36.995848 7fc5b9b29700 5 -- op tracker -- seq: 47408, time: 2014-11-08 18:15:36.995831, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -529> 2014-11-08 18:15:36.995851 7fc5b9b29700 5 -- op tracker -- seq: 47408, time: 2014-11-08 18:15:36.995851, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -528> 2014-11-08 18:15:36.995856 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -527> 2014-11-08 18:15:36.995862 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0dc0 -- 0x23724a40 -526> 2014-11-08 18:15:36.995879 7fc5b9b29700 5 -- op tracker -- seq: 47408, time: 2014-11-08 18:15:36.995879, event: done, op: pg_trim(4.c to 5734'8057 e5773) -525> 2014-11-08 18:15:36.995879 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23724a40 sd=135 :6811 s=4 pgs=8270 cs=69 l=0 c=0x23bb0dc0).reader couldn't read tag, (0) Success -524> 2014-11-08 18:15:36.995891 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234c380 con 0x23bb0dc0 -523> 2014-11-08 18:15:36.995892 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23724a40 sd=135 :6811 s=4 pgs=8270 cs=69 l=0 c=0x23bb0dc0).fault (0) Success -522> 2014-11-08 18:15:36.995898 7fc5b9b29700 5 -- op tracker -- seq: 47409, time: 2014-11-08 18:15:36.995770, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -521> 2014-11-08 18:15:36.995901 7fc5b9b29700 5 -- op tracker -- seq: 47409, time: 2014-11-08 18:15:36.995770, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -520> 2014-11-08 18:15:36.995903 7fc5b9b29700 5 -- op tracker -- seq: 47409, time: 2014-11-08 18:15:36.995790, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -519> 2014-11-08 18:15:36.995906 7fc5b9b29700 5 -- op tracker -- seq: 47409, time: 2014-11-08 18:15:36.995897, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -518> 2014-11-08 18:15:36.995909 7fc5b9b29700 5 -- op tracker -- seq: 47409, time: 2014-11-08 18:15:36.995909, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -517> 2014-11-08 18:15:36.995912 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -516> 2014-11-08 18:15:36.995921 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0dc0 -- pipe dne -515> 2014-11-08 18:15:36.995924 7fc5b9b29700 5 -- op tracker -- seq: 47409, time: 2014-11-08 18:15:36.995923, event: done, op: pg_trim(4.c to 5734'8057 e5773) -514> 2014-11-08 18:15:36.995930 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x2234a1c0 con 0x23bb0dc0 -513> 2014-11-08 18:15:36.995937 7fc5b9b29700 5 -- op tracker -- seq: 47410, time: 2014-11-08 18:15:36.995796, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -512> 2014-11-08 18:15:36.995943 7fc5b9b29700 5 -- op tracker -- seq: 47410, time: 2014-11-08 18:15:36.995796, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -511> 2014-11-08 18:15:36.995947 7fc5b9b29700 5 -- op tracker -- seq: 47410, time: 2014-11-08 18:15:36.995813, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -510> 2014-11-08 18:15:36.995951 7fc5b9b29700 5 -- op tracker -- seq: 47410, time: 2014-11-08 18:15:36.995936, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -509> 2014-11-08 18:15:36.995956 7fc5b9b29700 5 -- op tracker -- seq: 47410, time: 2014-11-08 18:15:36.995956, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -508> 2014-11-08 18:15:36.995959 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -507> 2014-11-08 18:15:36.995964 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0dc0 -- pipe dne -506> 2014-11-08 18:15:36.995968 7fc5b9b29700 5 -- op tracker -- seq: 47410, time: 2014-11-08 18:15:36.995968, event: done, op: pg_trim(4.c to 5734'8057 e5773) -505> 2014-11-08 18:15:36.998497 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23727bc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0000).accept sd=135 192.168.1.16:39211/0 -504> 2014-11-08 18:15:37.002607 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x1ea78b40 con 0x23bb0000 -503> 2014-11-08 18:15:37.002633 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -502> 2014-11-08 18:15:37.003975 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x22348000 con 0x23bb0000 -501> 2014-11-08 18:15:37.003988 7fc5b9b29700 5 -- op tracker -- seq: 47411, time: 2014-11-08 18:15:37.003920, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -500> 2014-11-08 18:15:37.003994 7fc5b9b29700 5 -- op tracker -- seq: 47411, time: 2014-11-08 18:15:37.003921, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -499> 2014-11-08 18:15:37.003996 7fc5b9b29700 5 -- op tracker -- seq: 47411, time: 2014-11-08 18:15:37.003964, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -498> 2014-11-08 18:15:37.003999 7fc5b9b29700 5 -- op tracker -- seq: 47411, time: 2014-11-08 18:15:37.003986, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -497> 2014-11-08 18:15:37.004002 7fc5b9b29700 5 -- op tracker -- seq: 47411, time: 2014-11-08 18:15:37.004002, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -496> 2014-11-08 18:15:37.004006 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -495> 2014-11-08 18:15:37.004010 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0000 -- 0x23727bc0 -494> 2014-11-08 18:15:37.004094 7fc5b9b29700 5 -- op tracker -- seq: 47411, time: 2014-11-08 18:15:37.004094, event: done, op: pg_trim(4.c to 5734'8057 e5773) -493> 2014-11-08 18:15:37.004107 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239edfa0 con 0x23bb0000 -492> 2014-11-08 18:15:37.004114 7fc5b9b29700 5 -- op tracker -- seq: 47412, time: 2014-11-08 18:15:37.004025, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -491> 2014-11-08 18:15:37.004117 7fc5b9b29700 5 -- op tracker -- seq: 47412, time: 2014-11-08 18:15:37.004026, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -490> 2014-11-08 18:15:37.004120 7fc5b9b29700 5 -- op tracker -- seq: 47412, time: 2014-11-08 18:15:37.004052, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -489> 2014-11-08 18:15:37.004122 7fc5b9b29700 5 -- op tracker -- seq: 47412, time: 2014-11-08 18:15:37.004113, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -488> 2014-11-08 18:15:37.004125 7fc5b9b29700 5 -- op tracker -- seq: 47412, time: 2014-11-08 18:15:37.004124, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -487> 2014-11-08 18:15:37.004127 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -486> 2014-11-08 18:15:37.004131 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0000 -- pipe dne -485> 2014-11-08 18:15:37.004133 7fc5b9b29700 5 -- op tracker -- seq: 47412, time: 2014-11-08 18:15:37.004133, event: done, op: pg_trim(4.c to 5734'8057 e5773) -484> 2014-11-08 18:15:37.004138 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x22349e00 con 0x23bb0000 -483> 2014-11-08 18:15:37.004144 7fc5b9b29700 5 -- op tracker -- seq: 47413, time: 2014-11-08 18:15:37.004057, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -482> 2014-11-08 18:15:37.004147 7fc5b9b29700 5 -- op tracker -- seq: 47413, time: 2014-11-08 18:15:37.004057, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -481> 2014-11-08 18:15:37.004149 7fc5b9b29700 5 -- op tracker -- seq: 47413, time: 2014-11-08 18:15:37.004074, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -480> 2014-11-08 18:15:37.004151 7fc5b9b29700 5 -- op tracker -- seq: 47413, time: 2014-11-08 18:15:37.004143, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -479> 2014-11-08 18:15:37.004154 7fc5b9b29700 5 -- op tracker -- seq: 47413, time: 2014-11-08 18:15:37.004154, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -478> 2014-11-08 18:15:37.004156 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -477> 2014-11-08 18:15:37.004159 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0000 -- pipe dne -476> 2014-11-08 18:15:37.004161 7fc5b9b29700 5 -- op tracker -- seq: 47413, time: 2014-11-08 18:15:37.004161, event: done, op: pg_trim(4.c to 5734'8057 e5773) -475> 2014-11-08 18:15:37.004179 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23727bc0 sd=135 :6811 s=4 pgs=8271 cs=71 l=0 c=0x23bb0000).do_sendmsg error (32) Broken pipe -474> 2014-11-08 18:15:37.004207 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23727bc0 sd=135 :6811 s=4 pgs=8271 cs=71 l=0 c=0x23bb0000).writer couldn't write ack, (32) Broken pipe -473> 2014-11-08 18:15:37.004221 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23727bc0 sd=135 :6811 s=4 pgs=8271 cs=71 l=0 c=0x23bb0000).fault (32) Broken pipe -472> 2014-11-08 18:15:37.004267 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23727bc0 sd=135 :6811 s=4 pgs=8271 cs=71 l=0 c=0x23bb0000).reader couldn't read tag, (0) Success -471> 2014-11-08 18:15:37.004282 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23727bc0 sd=135 :6811 s=4 pgs=8271 cs=71 l=0 c=0x23bb0000).fault (0) Success -470> 2014-11-08 18:15:37.007536 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23723c80 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb39c0).accept sd=135 192.168.1.16:39212/0 -469> 2014-11-08 18:15:37.012425 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x23029200 con 0x23bb39c0 -468> 2014-11-08 18:15:37.012453 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -467> 2014-11-08 18:15:37.013775 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239ee540 con 0x23bb39c0 -466> 2014-11-08 18:15:37.013788 7fc5b9b29700 5 -- op tracker -- seq: 47414, time: 2014-11-08 18:15:37.013732, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -465> 2014-11-08 18:15:37.013795 7fc5b9b29700 5 -- op tracker -- seq: 47414, time: 2014-11-08 18:15:37.013733, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -464> 2014-11-08 18:15:37.013798 7fc5b9b29700 5 -- op tracker -- seq: 47414, time: 2014-11-08 18:15:37.013763, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -463> 2014-11-08 18:15:37.013802 7fc5b9b29700 5 -- op tracker -- seq: 47414, time: 2014-11-08 18:15:37.013786, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -462> 2014-11-08 18:15:37.013806 7fc5b9b29700 5 -- op tracker -- seq: 47414, time: 2014-11-08 18:15:37.013806, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -461> 2014-11-08 18:15:37.013812 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -460> 2014-11-08 18:15:37.013819 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb39c0 -- 0x23723c80 -459> 2014-11-08 18:15:37.013892 7fc5b9b29700 5 -- op tracker -- seq: 47414, time: 2014-11-08 18:15:37.013892, event: done, op: pg_trim(4.c to 5734'8057 e5773) -458> 2014-11-08 18:15:37.013906 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239eeae0 con 0x23bb39c0 -457> 2014-11-08 18:15:37.013915 7fc5b9b29700 5 -- op tracker -- seq: 47415, time: 2014-11-08 18:15:37.013829, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -456> 2014-11-08 18:15:37.013918 7fc5b9b29700 5 -- op tracker -- seq: 47415, time: 2014-11-08 18:15:37.013830, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -455> 2014-11-08 18:15:37.013920 7fc5b9b29700 5 -- op tracker -- seq: 47415, time: 2014-11-08 18:15:37.013850, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -454> 2014-11-08 18:15:37.013925 7fc5b9b29700 5 -- op tracker -- seq: 47415, time: 2014-11-08 18:15:37.013914, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -453> 2014-11-08 18:15:37.013932 7fc5b9b29700 5 -- op tracker -- seq: 47415, time: 2014-11-08 18:15:37.013932, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -452> 2014-11-08 18:15:37.013937 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -451> 2014-11-08 18:15:37.013942 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb39c0 -- pipe dne -450> 2014-11-08 18:15:37.013946 7fc5b9b29700 5 -- op tracker -- seq: 47415, time: 2014-11-08 18:15:37.013946, event: done, op: pg_trim(4.c to 5734'8057 e5773) -449> 2014-11-08 18:15:37.013954 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239ec560 con 0x23bb39c0 -448> 2014-11-08 18:15:37.013962 7fc5b9b29700 5 -- op tracker -- seq: 47416, time: 2014-11-08 18:15:37.013855, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -447> 2014-11-08 18:15:37.013966 7fc5b9b29700 5 -- op tracker -- seq: 47416, time: 2014-11-08 18:15:37.013855, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -446> 2014-11-08 18:15:37.013970 7fc5b9b29700 5 -- op tracker -- seq: 47416, time: 2014-11-08 18:15:37.013872, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -445> 2014-11-08 18:15:37.013972 7fc5b9b29700 5 -- op tracker -- seq: 47416, time: 2014-11-08 18:15:37.013961, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -444> 2014-11-08 18:15:37.013977 7fc5b9b29700 5 -- op tracker -- seq: 47416, time: 2014-11-08 18:15:37.013977, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -443> 2014-11-08 18:15:37.013980 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -442> 2014-11-08 18:15:37.013984 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb39c0 -- pipe dne -441> 2014-11-08 18:15:37.013987 7fc5b9b29700 5 -- op tracker -- seq: 47416, time: 2014-11-08 18:15:37.013987, event: done, op: pg_trim(4.c to 5734'8057 e5773) -440> 2014-11-08 18:15:37.013997 7fc5a28c1700 1 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23723c80 sd=135 :6811 s=4 pgs=8272 cs=73 l=0 c=0x23bb39c0).do_sendmsg error (32) Broken pipe -439> 2014-11-08 18:15:37.014026 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23723c80 sd=135 :6811 s=4 pgs=8272 cs=73 l=0 c=0x23bb39c0).writer couldn't write ack, (32) Broken pipe -438> 2014-11-08 18:15:37.014035 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23723c80 sd=135 :6811 s=4 pgs=8272 cs=73 l=0 c=0x23bb39c0).fault (32) Broken pipe -437> 2014-11-08 18:15:37.014070 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23723c80 sd=135 :6811 s=4 pgs=8272 cs=73 l=0 c=0x23bb39c0).reader couldn't read tag, (0) Success -436> 2014-11-08 18:15:37.014080 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23723c80 sd=135 :6811 s=4 pgs=8272 cs=73 l=0 c=0x23bb39c0).fault (0) Success -435> 2014-11-08 18:15:37.017804 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x237270c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0160).accept sd=135 192.168.1.16:39213/0 -434> 2014-11-08 18:15:37.023925 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302de80 con 0x23bb0160 -433> 2014-11-08 18:15:37.023953 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -432> 2014-11-08 18:15:37.025994 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239eb840 con 0x23bb0160 -431> 2014-11-08 18:15:37.026009 7fc5b9b29700 5 -- op tracker -- seq: 47417, time: 2014-11-08 18:15:37.025895, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -430> 2014-11-08 18:15:37.026018 7fc5b9b29700 5 -- op tracker -- seq: 47417, time: 2014-11-08 18:15:37.025896, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -429> 2014-11-08 18:15:37.026021 7fc5b9b29700 5 -- op tracker -- seq: 47417, time: 2014-11-08 18:15:37.025934, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -428> 2014-11-08 18:15:37.026025 7fc5b9b29700 5 -- op tracker -- seq: 47417, time: 2014-11-08 18:15:37.026007, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -427> 2014-11-08 18:15:37.026028 7fc5b9b29700 5 -- op tracker -- seq: 47417, time: 2014-11-08 18:15:37.026028, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -426> 2014-11-08 18:15:37.026034 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -425> 2014-11-08 18:15:37.026040 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0160 -- 0x237270c0 -424> 2014-11-08 18:15:37.026055 7fc5b9b29700 5 -- op tracker -- seq: 47417, time: 2014-11-08 18:15:37.026055, event: done, op: pg_trim(4.c to 5734'8057 e5773) -423> 2014-11-08 18:15:37.026067 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239eb660 con 0x23bb0160 -422> 2014-11-08 18:15:37.026075 7fc5b9b29700 5 -- op tracker -- seq: 47418, time: 2014-11-08 18:15:37.025944, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -421> 2014-11-08 18:15:37.026080 7fc5b9b29700 5 -- op tracker -- seq: 47418, time: 2014-11-08 18:15:37.025945, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -420> 2014-11-08 18:15:37.026083 7fc5b9b29700 5 -- op tracker -- seq: 47418, time: 2014-11-08 18:15:37.025964, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -419> 2014-11-08 18:15:37.026087 7fc5b9b29700 5 -- op tracker -- seq: 47418, time: 2014-11-08 18:15:37.026074, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -418> 2014-11-08 18:15:37.026090 7fc5b9b29700 5 -- op tracker -- seq: 47418, time: 2014-11-08 18:15:37.026090, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -417> 2014-11-08 18:15:37.026094 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -416> 2014-11-08 18:15:37.026098 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0160 -- pipe dne -415> 2014-11-08 18:15:37.026100 7fc5b9b29700 5 -- op tracker -- seq: 47418, time: 2014-11-08 18:15:37.026100, event: done, op: pg_trim(4.c to 5734'8057 e5773) -414> 2014-11-08 18:15:37.026108 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239ed640 con 0x23bb0160 -413> 2014-11-08 18:15:37.026116 7fc5b9b29700 5 -- op tracker -- seq: 47419, time: 2014-11-08 18:15:37.025969, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -412> 2014-11-08 18:15:37.026119 7fc5b9b29700 5 -- op tracker -- seq: 47419, time: 2014-11-08 18:15:37.025969, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -411> 2014-11-08 18:15:37.026122 7fc5b9b29700 5 -- op tracker -- seq: 47419, time: 2014-11-08 18:15:37.025986, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -410> 2014-11-08 18:15:37.026126 7fc5b9b29700 5 -- op tracker -- seq: 47419, time: 2014-11-08 18:15:37.026115, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -409> 2014-11-08 18:15:37.026130 7fc5b9b29700 5 -- op tracker -- seq: 47419, time: 2014-11-08 18:15:37.026129, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -408> 2014-11-08 18:15:37.026134 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -407> 2014-11-08 18:15:37.026137 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0160 -- pipe dne -406> 2014-11-08 18:15:37.026142 7fc5b9b29700 5 -- op tracker -- seq: 47419, time: 2014-11-08 18:15:37.026142, event: done, op: pg_trim(4.c to 5734'8057 e5773) -405> 2014-11-08 18:15:37.026151 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x237270c0 sd=135 :6811 s=4 pgs=8273 cs=75 l=0 c=0x23bb0160).reader couldn't read tag, (0) Success -404> 2014-11-08 18:15:37.026164 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x237270c0 sd=135 :6811 s=4 pgs=8273 cs=75 l=0 c=0x23bb0160).fault (0) Success -403> 2014-11-08 18:15:37.028694 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.10 192.168.1.18:0/1404 3752 ==== osd_ping(ping e5776 stamp 2014-11-08 18:15:37.028149) v2 ==== 47+0+0 (2401245697 0 0) 0x2234d460 con 0x1cce3180 -402> 2014-11-08 18:15:37.028715 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5776 stamp 2014-11-08 18:15:37.028149) v2 -- ?+0 0x1caa8960 con 0x1cce3180 -401> 2014-11-08 18:15:37.028733 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.10 192.168.1.18:0/1404 3752 ==== osd_ping(ping e5776 stamp 2014-11-08 18:15:37.028149) v2 ==== 47+0+0 (2401245697 0 0) 0x2127dfa0 con 0x1cce3700 -400> 2014-11-08 18:15:37.028742 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.18:0/1404 -- osd_ping(ping_reply e5776 stamp 2014-11-08 18:15:37.028149) v2 -- ?+0 0x23682d00 con 0x1cce3700 -399> 2014-11-08 18:15:37.030569 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23726040 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb5d80).accept sd=135 192.168.1.16:39215/0 -398> 2014-11-08 18:15:37.038640 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302c800 con 0x23bb5d80 -397> 2014-11-08 18:15:37.038668 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -396> 2014-11-08 18:15:37.040854 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239ef260 con 0x23bb5d80 -395> 2014-11-08 18:15:37.040869 7fc5b9b29700 5 -- op tracker -- seq: 47420, time: 2014-11-08 18:15:37.040811, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -394> 2014-11-08 18:15:37.040877 7fc5b9b29700 5 -- op tracker -- seq: 47420, time: 2014-11-08 18:15:37.040813, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -393> 2014-11-08 18:15:37.040881 7fc5b9b29700 5 -- op tracker -- seq: 47420, time: 2014-11-08 18:15:37.040844, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -392> 2014-11-08 18:15:37.040886 7fc5b9b29700 5 -- op tracker -- seq: 47420, time: 2014-11-08 18:15:37.040867, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -391> 2014-11-08 18:15:37.040890 7fc5b9b29700 5 -- op tracker -- seq: 47420, time: 2014-11-08 18:15:37.040890, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -390> 2014-11-08 18:15:37.040897 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -389> 2014-11-08 18:15:37.040903 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb5d80 -- 0x23726040 -388> 2014-11-08 18:15:37.040919 7fc5b9b29700 5 -- op tracker -- seq: 47420, time: 2014-11-08 18:15:37.040918, event: done, op: pg_trim(4.c to 5734'8057 e5773) -387> 2014-11-08 18:15:37.040931 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239efda0 con 0x23bb5d80 -386> 2014-11-08 18:15:37.040939 7fc5b9b29700 5 -- op tracker -- seq: 47421, time: 2014-11-08 18:15:37.040854, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -385> 2014-11-08 18:15:37.040946 7fc5b9b29700 5 -- op tracker -- seq: 47421, time: 2014-11-08 18:15:37.040855, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -384> 2014-11-08 18:15:37.040950 7fc5b9b29700 5 -- op tracker -- seq: 47421, time: 2014-11-08 18:15:37.040900, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -383> 2014-11-08 18:15:37.040919 7fc5a39d2700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x23726040 sd=135 :6811 s=4 pgs=8275 cs=77 l=0 c=0x23bb5d80).fault (0) Success -382> 2014-11-08 18:15:37.040959 7fc5b9b29700 5 -- op tracker -- seq: 47421, time: 2014-11-08 18:15:37.040938, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -381> 2014-11-08 18:15:37.040969 7fc5b9b29700 5 -- op tracker -- seq: 47421, time: 2014-11-08 18:15:37.040969, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -380> 2014-11-08 18:15:37.040974 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -379> 2014-11-08 18:15:37.040979 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb5d80 -- pipe dne -378> 2014-11-08 18:15:37.040984 7fc5b9b29700 5 -- op tracker -- seq: 47421, time: 2014-11-08 18:15:37.040984, event: done, op: pg_trim(4.c to 5734'8057 e5773) -377> 2014-11-08 18:15:37.046432 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23725ac0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb1600).accept sd=135 192.168.1.16:39218/0 -376> 2014-11-08 18:15:37.055362 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302e540 con 0x23bb1600 -375> 2014-11-08 18:15:37.055391 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -374> 2014-11-08 18:15:37.058256 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 5 ==== MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (503779617 0 0) 0x2302ee40 con 0x23bb1600 -373> 2014-11-08 18:15:37.058275 7fc5a39d2700 5 -- op tracker -- seq: 47422, time: 2014-11-08 18:15:37.058217, event: header_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -372> 2014-11-08 18:15:37.058284 7fc5a39d2700 5 -- op tracker -- seq: 47422, time: 2014-11-08 18:15:37.058217, event: throttled, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -371> 2014-11-08 18:15:37.058290 7fc5a39d2700 5 -- op tracker -- seq: 47422, time: 2014-11-08 18:15:37.058253, event: all_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -370> 2014-11-08 18:15:37.058296 7fc5a39d2700 5 -- op tracker -- seq: 47422, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -369> 2014-11-08 18:15:37.058305 7fc5a39d2700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -368> 2014-11-08 18:15:37.058309 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb1600 -- 0x23725ac0 -367> 2014-11-08 18:15:37.058323 7fc5a39d2700 5 -- op tracker -- seq: 47422, time: 2014-11-08 18:15:37.058323, event: done, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -366> 2014-11-08 18:15:37.058371 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239eab20 con 0x23bb1600 -365> 2014-11-08 18:15:37.058380 7fc5b9b29700 5 -- op tracker -- seq: 47423, time: 2014-11-08 18:15:37.058101, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -364> 2014-11-08 18:15:37.058383 7fc5b9b29700 5 -- op tracker -- seq: 47423, time: 2014-11-08 18:15:37.058103, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -363> 2014-11-08 18:15:37.058386 7fc5b9b29700 5 -- op tracker -- seq: 47423, time: 2014-11-08 18:15:37.058154, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -362> 2014-11-08 18:15:37.058388 7fc5b9b29700 5 -- op tracker -- seq: 47423, time: 2014-11-08 18:15:37.058379, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -361> 2014-11-08 18:15:37.058391 7fc5b9b29700 5 -- op tracker -- seq: 47423, time: 2014-11-08 18:15:37.058391, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -360> 2014-11-08 18:15:37.058394 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -359> 2014-11-08 18:15:37.058398 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb1600 -- pipe dne -358> 2014-11-08 18:15:37.058401 7fc5b9b29700 5 -- op tracker -- seq: 47423, time: 2014-11-08 18:15:37.058401, event: done, op: pg_trim(4.c to 5734'8057 e5773) -357> 2014-11-08 18:15:37.058407 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239ef440 con 0x23bb1600 -356> 2014-11-08 18:15:37.058413 7fc5b9b29700 5 -- op tracker -- seq: 47424, time: 2014-11-08 18:15:37.058165, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -355> 2014-11-08 18:15:37.058416 7fc5b9b29700 5 -- op tracker -- seq: 47424, time: 2014-11-08 18:15:37.058166, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -354> 2014-11-08 18:15:37.058418 7fc5b9b29700 5 -- op tracker -- seq: 47424, time: 2014-11-08 18:15:37.058185, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -353> 2014-11-08 18:15:37.058421 7fc5b9b29700 5 -- op tracker -- seq: 47424, time: 2014-11-08 18:15:37.058412, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -352> 2014-11-08 18:15:37.058424 7fc5b9b29700 5 -- op tracker -- seq: 47424, time: 2014-11-08 18:15:37.058424, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -351> 2014-11-08 18:15:37.058426 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -350> 2014-11-08 18:15:37.058429 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb1600 -- pipe dne -349> 2014-11-08 18:15:37.058432 7fc5b9b29700 5 -- op tracker -- seq: 47424, time: 2014-11-08 18:15:37.058432, event: done, op: pg_trim(4.c to 5734'8057 e5773) -348> 2014-11-08 18:15:37.058437 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 4 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239ee900 con 0x23bb1600 -347> 2014-11-08 18:15:37.058442 7fc5b9b29700 5 -- op tracker -- seq: 47425, time: 2014-11-08 18:15:37.058191, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -346> 2014-11-08 18:15:37.058445 7fc5b9b29700 5 -- op tracker -- seq: 47425, time: 2014-11-08 18:15:37.058192, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -345> 2014-11-08 18:15:37.058448 7fc5b9b29700 5 -- op tracker -- seq: 47425, time: 2014-11-08 18:15:37.058209, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -344> 2014-11-08 18:15:37.058450 7fc5b9b29700 5 -- op tracker -- seq: 47425, time: 2014-11-08 18:15:37.058442, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -343> 2014-11-08 18:15:37.058453 7fc5b9b29700 5 -- op tracker -- seq: 47425, time: 2014-11-08 18:15:37.058453, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -342> 2014-11-08 18:15:37.058455 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -341> 2014-11-08 18:15:37.058458 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb1600 -- pipe dne -340> 2014-11-08 18:15:37.058461 7fc5b9b29700 5 -- op tracker -- seq: 47425, time: 2014-11-08 18:15:37.058461, event: done, op: pg_trim(4.c to 5734'8057 e5773) -339> 2014-11-08 18:15:37.063690 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x23724fc0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb02c0).accept sd=135 192.168.1.16:39220/0 -338> 2014-11-08 18:15:37.073075 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302c140 con 0x23bb02c0 -337> 2014-11-08 18:15:37.073095 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -336> 2014-11-08 18:15:37.075872 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239e8f00 con 0x23bb02c0 -335> 2014-11-08 18:15:37.075885 7fc5b9b29700 5 -- op tracker -- seq: 47426, time: 2014-11-08 18:15:37.075817, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -334> 2014-11-08 18:15:37.075891 7fc5b9b29700 5 -- op tracker -- seq: 47426, time: 2014-11-08 18:15:37.075818, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -333> 2014-11-08 18:15:37.075893 7fc5b9b29700 5 -- op tracker -- seq: 47426, time: 2014-11-08 18:15:37.075857, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -332> 2014-11-08 18:15:37.075896 7fc5b9b29700 5 -- op tracker -- seq: 47426, time: 2014-11-08 18:15:37.075884, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -331> 2014-11-08 18:15:37.075899 7fc5b9b29700 5 -- op tracker -- seq: 47426, time: 2014-11-08 18:15:37.075899, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -330> 2014-11-08 18:15:37.075903 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -329> 2014-11-08 18:15:37.075908 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb02c0 -- 0x23724fc0 -328> 2014-11-08 18:15:37.075923 7fc5b9b29700 5 -- op tracker -- seq: 47426, time: 2014-11-08 18:15:37.075923, event: done, op: pg_trim(4.c to 5734'8057 e5773) -327> 2014-11-08 18:15:37.075933 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 3 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239edbe0 con 0x23bb02c0 -326> 2014-11-08 18:15:37.075939 7fc5b9b29700 5 -- op tracker -- seq: 47427, time: 2014-11-08 18:15:37.075874, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -325> 2014-11-08 18:15:37.075942 7fc5b9b29700 5 -- op tracker -- seq: 47427, time: 2014-11-08 18:15:37.075874, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -324> 2014-11-08 18:15:37.075944 7fc5b9b29700 5 -- op tracker -- seq: 47427, time: 2014-11-08 18:15:37.075893, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -323> 2014-11-08 18:15:37.075947 7fc5b9b29700 5 -- op tracker -- seq: 47427, time: 2014-11-08 18:15:37.075938, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -322> 2014-11-08 18:15:37.075950 7fc5b9b29700 5 -- op tracker -- seq: 47427, time: 2014-11-08 18:15:37.075950, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -321> 2014-11-08 18:15:37.075953 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -320> 2014-11-08 18:15:37.075956 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb02c0 -- pipe dne -319> 2014-11-08 18:15:37.075958 7fc5b9b29700 5 -- op tracker -- seq: 47427, time: 2014-11-08 18:15:37.075958, event: done, op: pg_trim(4.c to 5734'8057 e5773) -318> 2014-11-08 18:15:37.080193 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 >> :/0 pipe(0x237239c0 sd=135 :6811 s=0 pgs=0 cs=0 l=0 c=0x23bb0420).accept sd=135 192.168.1.16:39222/0 -317> 2014-11-08 18:15:37.081372 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 1 ==== osd_map(5774..5774 src has 4761..5774) v3 ==== 211+0+0 (204081883 0 0) 0x2302ee40 con 0x23bb0420 -316> 2014-11-08 18:15:37.081391 7fc5b9b29700 3 osd.9 5776 handle_osd_map epochs [5774,5774], i have 5776, src has [4761,5774] -315> 2014-11-08 18:15:37.081468 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 2 ==== pg_trim(4.c to 5734'8057 e5773) v1 ==== 34+0+0 (702970203 0 0) 0x239e9e00 con 0x23bb0420 -314> 2014-11-08 18:15:37.081478 7fc5b9b29700 5 -- op tracker -- seq: 47428, time: 2014-11-08 18:15:37.081434, event: header_read, op: pg_trim(4.c to 5734'8057 e5773) -313> 2014-11-08 18:15:37.081484 7fc5b9b29700 5 -- op tracker -- seq: 47428, time: 2014-11-08 18:15:37.081435, event: throttled, op: pg_trim(4.c to 5734'8057 e5773) -312> 2014-11-08 18:15:37.081486 7fc5b9b29700 5 -- op tracker -- seq: 47428, time: 2014-11-08 18:15:37.081457, event: all_read, op: pg_trim(4.c to 5734'8057 e5773) -311> 2014-11-08 18:15:37.081489 7fc5b9b29700 5 -- op tracker -- seq: 47428, time: 2014-11-08 18:15:37.081477, event: dispatched, op: pg_trim(4.c to 5734'8057 e5773) -310> 2014-11-08 18:15:37.081492 7fc5b9b29700 5 -- op tracker -- seq: 47428, time: 2014-11-08 18:15:37.081491, event: waiting_for_osdmap, op: pg_trim(4.c to 5734'8057 e5773) -309> 2014-11-08 18:15:37.081496 7fc5b9b29700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -308> 2014-11-08 18:15:37.081501 7fc5b9b29700 1 -- 192.168.1.18:6811/4928 mark_down 0x23bb0420 -- 0x237239c0 -307> 2014-11-08 18:15:37.081621 7fc5a39d2700 1 -- 192.168.1.18:6811/4928 <== osd.4 192.168.1.16:6815/6031236 5 ==== MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (503779617 0 0) 0x23029f80 con 0x23bb0420 -306> 2014-11-08 18:15:37.081639 7fc5a39d2700 5 -- op tracker -- seq: 47429, time: 2014-11-08 18:15:37.081578, event: header_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -305> 2014-11-08 18:15:37.081647 7fc5a39d2700 5 -- op tracker -- seq: 47429, time: 2014-11-08 18:15:37.081578, event: throttled, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -304> 2014-11-08 18:15:37.081655 7fc5a39d2700 5 -- op tracker -- seq: 47429, time: 2014-11-08 18:15:37.081620, event: all_read, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -303> 2014-11-08 18:15:37.081661 7fc5a39d2700 5 -- op tracker -- seq: 47429, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.8fs1 5773 ECSubRead(tid=59255, to_read={f520778f/1000002b9bb.0000013b/head//4=0,4194304,5fb0778f/1000002ad50.0000006e/head//4=0,4194304,7871778f/100000199c7.0000027a/head//4=0,4194304,8b91778f/10000019bde.00000734/head//4=0,4194304,b502778f/10000022aef.00000000/head//4=0,4194304}, attrs_to_read=)) -302> 2014-11-08 18:15:37.081669 7fc5a39d2700 5 osd.9 5776 from dead osd.4, marking down, msg was 192.168.1.16:6815/6031236 expected :/0 -301> 2014-11-08 18:15:37.081736 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x237239c0 sd=135 :6811 s=4 pgs=8282 cs=83 l=0 c=0x23bb0420).writer couldn't write ack, (4) Interrupted system call -300> 2014-11-08 18:15:37.081757 7fc5a28c1700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6815/6031236 pipe(0x237239c0 sd=135 :6811 s=4 pgs=8282 cs=83 l=0 c=0x23bb0420).fault (4) Interrupted system call -299> 2014-11-08 18:15:37.491707 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8655 ==== pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) v3 ==== 757+0+0 (3105505913 0 0) 0x23ceec00 con 0x1c5c0580 -298> 2014-11-08 18:15:37.491744 7fc5a152e700 5 -- op tracker -- seq: 47430, time: 2014-11-08 18:15:37.491596, event: header_read, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -297> 2014-11-08 18:15:37.491753 7fc5a152e700 5 -- op tracker -- seq: 47430, time: 2014-11-08 18:15:37.491602, event: throttled, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -296> 2014-11-08 18:15:37.491756 7fc5a152e700 5 -- op tracker -- seq: 47430, time: 2014-11-08 18:15:37.491698, event: all_read, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -295> 2014-11-08 18:15:37.491759 7fc5a152e700 5 -- op tracker -- seq: 47430, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -294> 2014-11-08 18:15:37.491846 7fc5afb15700 5 -- op tracker -- seq: 47430, time: 2014-11-08 18:15:37.491846, event: reached_pg, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -293> 2014-11-08 18:15:37.491856 7fc5afb15700 5 -- op tracker -- seq: 47430, time: 2014-11-08 18:15:37.491856, event: started, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -292> 2014-11-08 18:15:37.491911 7fc5afb15700 5 -- op tracker -- seq: 47430, time: 2014-11-08 18:15:37.491911, event: done, op: pg_backfill(progress 4.16s2 e 5774/5774 lb 5c5e9b16/1000001a265.0000005f/head//4) -291> 2014-11-08 18:15:37.503700 7fc5b7324700 1 -- 192.168.1.18:6813/4928 <== osd.5 192.168.1.16:0/1550 327 ==== osd_ping(ping e5776 stamp 2014-11-08 18:15:37.501790) v2 ==== 47+0+0 (702639583 0 0) 0x224a6900 con 0x1eb9c780 -290> 2014-11-08 18:15:37.503741 7fc5b7324700 1 -- 192.168.1.18:6813/4928 --> 192.168.1.16:0/1550 -- osd_ping(ping_reply e5776 stamp 2014-11-08 18:15:37.501790) v2 -- ?+0 0x2127dfa0 con 0x1eb9c780 -289> 2014-11-08 18:15:37.503766 7fc5b5b21700 1 -- 192.168.1.18:6812/4928 <== osd.5 192.168.1.16:0/1550 327 ==== osd_ping(ping e5776 stamp 2014-11-08 18:15:37.501790) v2 ==== 47+0+0 (702639583 0 0) 0x224a6540 con 0x1eb9d6a0 -288> 2014-11-08 18:15:38.483208 7fc5a27c0700 1 -- 192.168.1.18:6813/4928 >> :/0 pipe(0x215a7900 sd=157 :6813 s=0 pgs=0 cs=0 l=0 c=0x1eb9b020).accept sd=157 192.168.1.16:54805/0 -287> 2014-11-08 18:15:38.483251 7fc5a2ac3700 1 -- 192.168.1.18:6812/4928 >> :/0 pipe(0x1d39b440 sd=158 :6812 s=0 pgs=0 cs=0 l=0 c=0x1eb9d3e0).accept sd=158 192.168.1.16:47749/0 -286> 2014-11-08 18:15:39.922668 7fc5b2b1b700 10 monclient: tick -285> 2014-11-08 18:15:39.922696 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:15:09.922695) -284> 2014-11-08 18:15:39.922706 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:15:39.922706; renew after: 2014-11-08 18:16:29.290049) -- no -283> 2014-11-08 18:15:40.021445 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8672 ==== MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 8446236+0+0 (2256607842 0 0) 0x1d3af400 con 0x1c5c0580 -282> 2014-11-08 18:15:40.021500 7fc5a152e700 5 -- op tracker -- seq: 47431, time: 2014-11-08 18:15:39.836387, event: header_read, op: MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -281> 2014-11-08 18:15:40.021525 7fc5a152e700 5 -- op tracker -- seq: 47431, time: 2014-11-08 18:15:39.836390, event: throttled, op: MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -280> 2014-11-08 18:15:40.021544 7fc5a152e700 5 -- op tracker -- seq: 47431, time: 2014-11-08 18:15:40.021434, event: all_read, op: MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -279> 2014-11-08 18:15:40.021565 7fc5a152e700 5 -- op tracker -- seq: 47431, time: 0.000000, event: dispatched, op: MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -278> 2014-11-08 18:15:40.021618 7fc5ad310700 5 -- op tracker -- seq: 47431, time: 2014-11-08 18:15:40.021618, event: reached_pg, op: MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -277> 2014-11-08 18:15:40.021647 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8673 ==== pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) v3 ==== 757+0+0 (1255065559 0 0) 0x1d3cd100 con 0x1c5c0580 -276> 2014-11-08 18:15:40.021655 7fc5a152e700 5 -- op tracker -- seq: 47432, time: 2014-11-08 18:15:40.021613, event: header_read, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -275> 2014-11-08 18:15:40.021659 7fc5a152e700 5 -- op tracker -- seq: 47432, time: 2014-11-08 18:15:40.021614, event: throttled, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -274> 2014-11-08 18:15:40.021662 7fc5a152e700 5 -- op tracker -- seq: 47432, time: 2014-11-08 18:15:40.021644, event: all_read, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -273> 2014-11-08 18:15:40.021665 7fc5a152e700 5 -- op tracker -- seq: 47432, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -272> 2014-11-08 18:15:40.021728 7fc5afb15700 5 -- op tracker -- seq: 47432, time: 2014-11-08 18:15:40.021726, event: reached_pg, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -271> 2014-11-08 18:15:40.021742 7fc5afb15700 5 -- op tracker -- seq: 47432, time: 2014-11-08 18:15:40.021742, event: started, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -270> 2014-11-08 18:15:40.021830 7fc5ad310700 5 -- op tracker -- seq: 47431, time: 2014-11-08 18:15:40.021830, event: done, op: MOSDPGPush(4.16s2 5776 [PushOp(6ade9b16/1000001a6cf.0000000d/head//4, version: 397'2100, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6ade9b16/1000001a6cf.0000000d/head//4@397'2100, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(f0ee9b16/1000002b1c0.0000000d/head//4, version: 4157'8085, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(f0ee9b16/1000002b1c0.0000000d/head//4@4157'8085, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(6110ab16/1000001956d.0000001a/head//4, version: 397'662, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6110ab16/1000001956d.0000001a/head//4@397'662, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(b890ab16/100000140cc.00000003/head//4, version: 397'384, data_included: [0~53248], data_size: 53248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(b890ab16/ 100000140cc.00000003/head//4@397'384, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:106496, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(c1d0ab16/1000002a703.00000003/head//4, version: 3458'5283, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(c1d0ab16/1000002a703.00000003/head//4@3458'5283, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -269> 2014-11-08 18:15:40.021919 7fc5a152e700 5 -- op tracker -- seq: 47432, time: 2014-11-08 18:15:40.021919, event: done, op: pg_backfill(progress 4.89s2 e 5776/5776 lb 17830589/1000002b5b0.0000003b/head//4) -268> 2014-11-08 18:15:40.022454 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8676 ==== MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) v2 ==== 175+0+0 (511482060 0 0) 0x229be000 con 0x1c5c0580 -267> 2014-11-08 18:15:40.022468 7fc5a152e700 5 -- op tracker -- seq: 47433, time: 2014-11-08 18:15:40.022028, event: header_read, op: MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) -266> 2014-11-08 18:15:40.022473 7fc5a152e700 5 -- op tracker -- seq: 47433, time: 2014-11-08 18:15:40.022029, event: throttled, op: MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) -265> 2014-11-08 18:15:40.022478 7fc5a152e700 5 -- op tracker -- seq: 47433, time: 2014-11-08 18:15:40.022451, event: all_read, op: MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) -264> 2014-11-08 18:15:40.022483 7fc5a152e700 5 -- op tracker -- seq: 47433, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) -263> 2014-11-08 18:15:40.022506 7fc5afb15700 5 -- op tracker -- seq: 47433, time: 2014-11-08 18:15:40.022506, event: reached_pg, op: MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) -262> 2014-11-08 18:15:40.022577 7fc5a152e700 5 -- op tracker -- seq: 47433, time: 2014-11-08 18:15:40.022577, event: done, op: MOSDPGPushReply(4.43s0 5776 [PushReplyOp(5981f343/1000002ab09.00000020/head//4),PushReplyOp(d7c1f343/1000002aa96.00000035/head//4)]) -261> 2014-11-08 18:15:41.193191 7fc5a152e700 1 -- 192.168.1.18:6811/4928 <== osd.2 192.168.1.15:6811/1606 8687 ==== MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87150589/ 1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 10490140+0+0 (3732969958 0 0) 0x229be000 con 0x1c5c0580 -260> 2014-11-08 18:15:41.193253 7fc5a152e700 5 -- op tracker -- seq: 47434, time: 2014-11-08 18:15:41.003568, event: header_read, op: MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87150589/1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -259> 2014-11-08 18:15:41.193284 7fc5a152e700 5 -- op tracker -- seq: 47434, time: 2014-11-08 18:15:41.003573, event: throttled, op: MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo( 87150589/1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -258> 2014-11-08 18:15:41.193309 7fc5a152e700 5 -- op tracker -- seq: 47434, time: 2014-11-08 18:15:41.193161, event: all_read, op: MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87150589/ 1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -257> 2014-11-08 18:15:41.193334 7fc5a152e700 5 -- op tracker -- seq: 47434, time: 0.000000, event: dispatched, op: MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87150589/ 1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -256> 2014-11-08 18:15:41.193397 7fc5ad310700 5 -- op tracker -- seq: 47434, time: 2014-11-08 18:15:41.193397, event: reached_pg, op: MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_ progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87150589/1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -255> 2014-11-08 18:15:41.193580 7fc5ad310700 5 -- op tracker -- seq: 47434, time: 2014-11-08 18:15:41.193579, event: done, op: MOSDPGPush(4.89s2 5776 [PushOp(ceb30589/1000002b72c.0000002f/head//4, version: 3473'6976, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(ceb30589/1000002b72c.0000002f/head//4@3473'6976, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(9640589/1000002a727.00000039/head//4, version: 3458'5366, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(9640589/1000002a727.00000039/head//4@3458'5366, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(10940589/1000002b873.0000007e/head//4, version: 3473'7107, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(10940589/1000002b873.0000007e/head//4@3473'7107, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(87150589/1000002b423.0000001d/head//4, version: 3468'6521, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(87150589/ 1000002b423.0000001d/head//4@3468'6521, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true)),PushOp(5e50589/1000002a8be.0000000a/head//4, version: 3458'5591, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5e50589/1000002a8be.0000000a/head//4@3458'5591, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -254> 2014-11-08 18:15:42.181807 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2316 ==== MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) v2 ==== 110+0+0 (2363614390 0 0) 0x1d3af200 con 0x1e50aec0 -253> 2014-11-08 18:15:42.181830 7fc5a56ef700 5 -- op tracker -- seq: 47435, time: 2014-11-08 18:15:42.181716, event: header_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) -252> 2014-11-08 18:15:42.181836 7fc5a56ef700 5 -- op tracker -- seq: 47435, time: 2014-11-08 18:15:42.181719, event: throttled, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) -251> 2014-11-08 18:15:42.181840 7fc5a56ef700 5 -- op tracker -- seq: 47435, time: 2014-11-08 18:15:42.181801, event: all_read, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) -250> 2014-11-08 18:15:42.181843 7fc5a56ef700 5 -- op tracker -- seq: 47435, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) -249> 2014-11-08 18:15:42.181877 7fc5ad310700 5 -- op tracker -- seq: 47435, time: 2014-11-08 18:15:42.181877, event: reached_pg, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) -248> 2014-11-08 18:15:42.181949 7fc5ad310700 5 -- op tracker -- seq: 47435, time: 2014-11-08 18:15:42.181949, event: done, op: MOSDPGPushReply(4.84s0 5773 [PushReplyOp(c149e484/1000002a5cd.0000007d/head//4)]) -247> 2014-11-08 18:15:42.189636 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12147 ==== MOSDECSubOpRead(4.27s1 5776 ECSubRead(tid=25207, to_read={a4884227/10000028670.0000006a/head//4=0,4194304}, attrs_to_read=)) v1 ==== 140+0+0 (2472565765 0 0) 0x2287e300 con 0x1c5c2680 -246> 2014-11-08 18:15:42.189714 7fc5a6cfe700 5 -- op tracker -- seq: 47436, time: 2014-11-08 18:15:42.189543, event: header_read, op: MOSDECSubOpRead(4.27s1 5776 ECSubRead(tid=25207, to_read={a4884227/10000028670.0000006a/head//4=0,4194304}, attrs_to_read=)) -245> 2014-11-08 18:15:42.189723 7fc5a6cfe700 5 -- op tracker -- seq: 47436, time: 2014-11-08 18:15:42.189547, event: throttled, op: MOSDECSubOpRead(4.27s1 5776 ECSubRead(tid=25207, to_read={a4884227/10000028670.0000006a/head//4=0,4194304}, attrs_to_read=)) -244> 2014-11-08 18:15:42.189727 7fc5a6cfe700 5 -- op tracker -- seq: 47436, time: 2014-11-08 18:15:42.189626, event: all_read, op: MOSDECSubOpRead(4.27s1 5776 ECSubRead(tid=25207, to_read={a4884227/10000028670.0000006a/head//4=0,4194304}, attrs_to_read=)) -243> 2014-11-08 18:15:42.189730 7fc5a6cfe700 5 -- op tracker -- seq: 47436, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.27s1 5776 ECSubRead(tid=25207, to_read={a4884227/10000028670.0000006a/head//4=0,4194304}, attrs_to_read=)) -242> 2014-11-08 18:15:42.189768 7fc5aeb13700 5 -- op tracker -- seq: 47436, time: 2014-11-08 18:15:42.189767, event: reached_pg, op: MOSDECSubOpRead(4.27s1 5776 ECSubRead(tid=25207, to_read={a4884227/10000028670.0000006a/head//4=0,4194304}, attrs_to_read=)) -241> 2014-11-08 18:15:43.348014 7fc5bb32c700 1 -- 192.168.1.18:6800/4928 <== mon.2 192.168.1.18:6789/0 1513 ==== osd_map(5776..5777 src has 4761..5777) v3 ==== 1022+0+0 (577617265 0 0) 0x22ac8480 con 0x44ed6a0 -240> 2014-11-08 18:15:45.282586 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12167 ==== pg_backfill(progress 4.88s1 e 5778/5778 lb efa1fd88/1000002a8e8.00000068/head//4) v3 ==== 757+0+0 (2540603079 0 0) 0x1d33b180 con 0x1c5c2680 -239> 2014-11-08 18:15:45.282607 7fc5a6cfe700 5 -- op tracker -- seq: 47437, time: 2014-11-08 18:15:45.282508, event: header_read, op: pg_backfill(progress 4.88s1 e 5778/5778 lb efa1fd88/1000002a8e8.00000068/head//4) -238> 2014-11-08 18:15:45.282617 7fc5a6cfe700 5 -- op tracker -- seq: 47437, time: 2014-11-08 18:15:45.282511, event: throttled, op: pg_backfill(progress 4.88s1 e 5778/5778 lb efa1fd88/1000002a8e8.00000068/head//4) -237> 2014-11-08 18:15:45.282622 7fc5a6cfe700 5 -- op tracker -- seq: 47437, time: 2014-11-08 18:15:45.282579, event: all_read, op: pg_backfill(progress 4.88s1 e 5778/5778 lb efa1fd88/1000002a8e8.00000068/head//4) -236> 2014-11-08 18:15:45.282625 7fc5a6cfe700 5 -- op tracker -- seq: 47437, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.88s1 e 5778/5778 lb efa1fd88/1000002a8e8.00000068/head//4) -235> 2014-11-08 18:15:46.431260 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2318 ==== MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) v2 ==== 370+0+0 (98078762 0 0) 0x21611200 con 0x1e50aec0 -234> 2014-11-08 18:15:46.431293 7fc5a56ef700 5 -- op tracker -- seq: 47438, time: 2014-11-08 18:15:46.431230, event: header_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) -233> 2014-11-08 18:15:46.431303 7fc5a56ef700 5 -- op tracker -- seq: 47438, time: 2014-11-08 18:15:46.431230, event: throttled, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) -232> 2014-11-08 18:15:46.431308 7fc5a56ef700 5 -- op tracker -- seq: 47438, time: 2014-11-08 18:15:46.431259, event: all_read, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) -231> 2014-11-08 18:15:46.431312 7fc5a56ef700 5 -- op tracker -- seq: 47438, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) -230> 2014-11-08 18:15:46.431373 7fc5ac30e700 5 -- op tracker -- seq: 47438, time: 2014-11-08 18:15:46.431373, event: reached_pg, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) -229> 2014-11-08 18:15:46.431489 7fc5ac30e700 5 -- op tracker -- seq: 47438, time: 2014-11-08 18:15:46.431489, event: done, op: MOSDPGPushReply(4.18s0 5773 [PushReplyOp(596d0518/10000019e0c.0000020a/head//4),PushReplyOp(5d8d0518/1000002a6bd.0000030e/head//4),PushReplyOp(50cd0518/10000019d87.00000225/head//4),PushReplyOp(176e0518/1000002bc6d.0000005b/head//4),PushReplyOp(f4ce0518/1000002a609.00000028/head//4)]) -228> 2014-11-08 18:15:46.482767 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2319 ==== pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) v2 ==== 139+0+0 (4148075473 0 0) 0x21a51b00 con 0x1e50aec0 -227> 2014-11-08 18:15:46.482801 7fc5a56ef700 5 -- op tracker -- seq: 47439, time: 2014-11-08 18:15:46.482651, event: header_read, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -226> 2014-11-08 18:15:46.482810 7fc5a56ef700 5 -- op tracker -- seq: 47439, time: 2014-11-08 18:15:46.482657, event: throttled, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -225> 2014-11-08 18:15:46.482813 7fc5a56ef700 5 -- op tracker -- seq: 47439, time: 2014-11-08 18:15:46.482760, event: all_read, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -224> 2014-11-08 18:15:46.482817 7fc5a56ef700 5 -- op tracker -- seq: 47439, time: 0.000000, event: dispatched, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -223> 2014-11-08 18:15:46.482859 7fc5afb15700 5 -- op tracker -- seq: 47439, time: 2014-11-08 18:15:46.482859, event: reached_pg, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -222> 2014-11-08 18:15:46.482870 7fc5afb15700 5 -- op tracker -- seq: 47439, time: 2014-11-08 18:15:46.482870, event: started, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -221> 2014-11-08 18:15:46.562895 7fc5afb15700 1 -- 192.168.1.18:6811/4928 --> 192.168.1.16:6806/12001550 -- pg_scan(digest 4.2fs1 b8d0202f/1000002b572.00000031/head//4-MAX e 5776/5776) v2 -- ?+359 0x2287de80 con 0x1e50aec0 -220> 2014-11-08 18:15:46.562929 7fc5afb15700 5 -- op tracker -- seq: 47439, time: 2014-11-08 18:15:46.562929, event: done, op: pg_scan(get_digest 4.2fs2 b8d0202f/1000002b572.00000031/head//4-0//0//4 e 5776/5776) -219> 2014-11-08 18:15:47.226595 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2320 ==== MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) v2 ==== 305+0+0 (137722071 0 0) 0x256fdc00 con 0x1e50aec0 -218> 2014-11-08 18:15:47.226634 7fc5a56ef700 5 -- op tracker -- seq: 47440, time: 2014-11-08 18:15:47.226489, event: header_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -217> 2014-11-08 18:15:47.226645 7fc5a56ef700 5 -- op tracker -- seq: 47440, time: 2014-11-08 18:15:47.226493, event: throttled, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -216> 2014-11-08 18:15:47.226650 7fc5a56ef700 5 -- op tracker -- seq: 47440, time: 2014-11-08 18:15:47.226586, event: all_read, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -215> 2014-11-08 18:15:47.226654 7fc5a56ef700 5 -- op tracker -- seq: 47440, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -214> 2014-11-08 18:15:47.226715 7fc5ac30e700 5 -- op tracker -- seq: 47440, time: 2014-11-08 18:15:47.226715, event: reached_pg, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -213> 2014-11-08 18:15:47.226833 7fc5ac30e700 5 -- op tracker -- seq: 47440, time: 2014-11-08 18:15:47.226833, event: done, op: MOSDPGPushReply(4.13s0 5773 [PushReplyOp(be8fdd13/1000002b2f6.00000008/head//4),PushReplyOp(e89fdd13/1000002b51d.0000000c/head//4),PushReplyOp(4cdfdd13/1000002982d.00000080/head//4),PushReplyOp(1d10ed13/10000019b94.00000084/head//4)]) -212> 2014-11-08 18:15:47.618925 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2321 ==== MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) v2 ==== 240+0+0 (2099924265 0 0) 0x2393c400 con 0x1e50aec0 -211> 2014-11-08 18:15:47.618962 7fc5a56ef700 5 -- op tracker -- seq: 47441, time: 2014-11-08 18:15:47.618821, event: header_read, op: MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) -210> 2014-11-08 18:15:47.618975 7fc5a56ef700 5 -- op tracker -- seq: 47441, time: 2014-11-08 18:15:47.618826, event: throttled, op: MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) -209> 2014-11-08 18:15:47.618980 7fc5a56ef700 5 -- op tracker -- seq: 47441, time: 2014-11-08 18:15:47.618917, event: all_read, op: MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) -208> 2014-11-08 18:15:47.618986 7fc5a56ef700 5 -- op tracker -- seq: 47441, time: 0.000000, event: dispatched, op: MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) -207> 2014-11-08 18:15:47.619047 7fc5b0316700 5 -- op tracker -- seq: 47441, time: 2014-11-08 18:15:47.619047, event: reached_pg, op: MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) -206> 2014-11-08 18:15:47.619162 7fc5b0316700 5 -- op tracker -- seq: 47441, time: 2014-11-08 18:15:47.619161, event: done, op: MOSDPGPushReply(4.83s0 5774 [PushReplyOp(5b9f1d83/1000002b56e.0000010b/head//4),PushReplyOp(86ff1d83/1000002b64a.00000041/head//4),PushReplyOp(42a02d83/1000002b040.00000026/head//4)]) -205> 2014-11-08 18:15:48.716680 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2332 ==== pg_backfill(progress 4.b7s1 e 5777/5777 lb a270a0b7/1000001987b.000000b3/head//4) v3 ==== 757+0+0 (2908246485 0 0) 0x1d3c9200 con 0x1e50aec0 -204> 2014-11-08 18:15:48.716713 7fc5a56ef700 5 -- op tracker -- seq: 47442, time: 2014-11-08 18:15:48.716603, event: header_read, op: pg_backfill(progress 4.b7s1 e 5777/5777 lb a270a0b7/1000001987b.000000b3/head//4) -203> 2014-11-08 18:15:48.716720 7fc5a56ef700 5 -- op tracker -- seq: 47442, time: 2014-11-08 18:15:48.716606, event: throttled, op: pg_backfill(progress 4.b7s1 e 5777/5777 lb a270a0b7/1000001987b.000000b3/head//4) -202> 2014-11-08 18:15:48.716723 7fc5a56ef700 5 -- op tracker -- seq: 47442, time: 2014-11-08 18:15:48.716674, event: all_read, op: pg_backfill(progress 4.b7s1 e 5777/5777 lb a270a0b7/1000001987b.000000b3/head//4) -201> 2014-11-08 18:15:48.716726 7fc5a56ef700 5 -- op tracker -- seq: 47442, time: 0.000000, event: dispatched, op: pg_backfill(progress 4.b7s1 e 5777/5777 lb a270a0b7/1000001987b.000000b3/head//4) -200> 2014-11-08 18:15:49.514998 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12185 ==== MOSDPGPush(4.88s1 5781 [PushOp(28e1fd88/1000002a36f.0000003e/head//4, version: 3033'4714, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(28e1fd88/1000002a36f.0000003e/head//4@3033'4714, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (885457015 0 0) 0x1d4c8400 con 0x1c5c2680 -199> 2014-11-08 18:15:49.515026 7fc5a6cfe700 5 -- op tracker -- seq: 47443, time: 2014-11-08 18:15:49.289700, event: header_read, op: MOSDPGPush(4.88s1 5781 [PushOp(28e1fd88/1000002a36f.0000003e/head//4, version: 3033'4714, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(28e1fd88/1000002a36f.0000003e/head//4@3033'4714, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -198> 2014-11-08 18:15:49.515036 7fc5a6cfe700 5 -- op tracker -- seq: 47443, time: 2014-11-08 18:15:49.289704, event: throttled, op: MOSDPGPush(4.88s1 5781 [PushOp(28e1fd88/1000002a36f.0000003e/head//4, version: 3033'4714, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(28e1fd88/1000002a36f.0000003e/head//4@3033'4714, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -197> 2014-11-08 18:15:49.515044 7fc5a6cfe700 5 -- op tracker -- seq: 47443, time: 2014-11-08 18:15:49.514991, event: all_read, op: MOSDPGPush(4.88s1 5781 [PushOp(28e1fd88/1000002a36f.0000003e/head//4, version: 3033'4714, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(28e1fd88/1000002a36f.0000003e/head//4@3033'4714, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -196> 2014-11-08 18:15:49.515052 7fc5a6cfe700 5 -- op tracker -- seq: 47443, time: 0.000000, event: dispatched, op: MOSDPGPush(4.88s1 5781 [PushOp(28e1fd88/1000002a36f.0000003e/head//4, version: 3033'4714, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(28e1fd88/1000002a36f.0000003e/head//4@3033'4714, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -195> 2014-11-08 18:15:49.922969 7fc5b2b1b700 10 monclient: tick -194> 2014-11-08 18:15:49.922988 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:15:19.922986) -193> 2014-11-08 18:15:49.923006 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:15:49.923005; renew after: 2014-11-08 18:16:29.290049) -- no -192> 2014-11-08 18:15:51.825410 7fc5a56ef700 1 -- 192.168.1.18:6811/4928 <== osd.5 192.168.1.16:6806/12001550 2334 ==== MOSDPGPush(4.b7s1 5781 [PushOp(5880a0b7/1000002b78c.00000059/head//4, version: 3473'6844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5880a0b7/1000002b78c.00000059/head//4@3473'6844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) v2 ==== 2098064+0+0 (4267329002 0 0) 0x256fe200 con 0x1e50aec0 -191> 2014-11-08 18:15:51.825459 7fc5a56ef700 5 -- op tracker -- seq: 47444, time: 2014-11-08 18:15:51.803665, event: header_read, op: MOSDPGPush(4.b7s1 5781 [PushOp(5880a0b7/1000002b78c.00000059/head//4, version: 3473'6844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5880a0b7/1000002b78c.00000059/head//4@3473'6844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -190> 2014-11-08 18:15:51.825477 7fc5a56ef700 5 -- op tracker -- seq: 47444, time: 2014-11-08 18:15:51.803669, event: throttled, op: MOSDPGPush(4.b7s1 5781 [PushOp(5880a0b7/1000002b78c.00000059/head//4, version: 3473'6844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5880a0b7/1000002b78c.00000059/head//4@3473'6844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -189> 2014-11-08 18:15:51.825490 7fc5a56ef700 5 -- op tracker -- seq: 47444, time: 2014-11-08 18:15:51.825382, event: all_read, op: MOSDPGPush(4.b7s1 5781 [PushOp(5880a0b7/1000002b78c.00000059/head//4, version: 3473'6844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5880a0b7/1000002b78c.00000059/head//4@3473'6844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -188> 2014-11-08 18:15:51.825503 7fc5a56ef700 5 -- op tracker -- seq: 47444, time: 0.000000, event: dispatched, op: MOSDPGPush(4.b7s1 5781 [PushOp(5880a0b7/1000002b78c.00000059/head//4, version: 3473'6844, data_included: [0~2097152], data_size: 2097152, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(5880a0b7/1000002b78c.00000059/head//4@3473'6844, copy_subset: [], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true))]) -187> 2014-11-08 18:15:52.056280 7fc5a6cfe700 1 -- 192.168.1.18:6811/4928 <== osd.1 192.168.1.15:6806/1411 12186 ==== MOSDECSubOpRead(4.bbs2 5781 ECSubRead(tid=25213, to_read={5c25a0bb/1000002b731.000000a6/head//4=0,4194304,b485a0bb/10000012c37.00000000/head//4=0,4194304,cea5a0bb/1000002b078.000001e0/head//4=0,4194304,4726a0bb/10000029591.00000016/head//4=0,4194304,2576a0bb/1000002a767.00000196/head//4=0,4194304}, attrs_to_read=)) v1 ==== 456+0+0 (3890730962 0 0) 0x22accc80 con 0x1c5c2680 -186> 2014-11-08 18:15:52.056320 7fc5a6cfe700 5 -- op tracker -- seq: 47445, time: 2014-11-08 18:15:52.056162, event: header_read, op: MOSDECSubOpRead(4.bbs2 5781 ECSubRead(tid=25213, to_read={5c25a0bb/1000002b731.000000a6/head//4=0,4194304,b485a0bb/10000012c37.00000000/head//4=0,4194304,cea5a0bb/1000002b078.000001e0/head//4=0,4194304,4726a0bb/10000029591.00000016/head//4=0,4194304,2576a0bb/1000002a767.00000196/head//4=0,4194304}, attrs_to_read=)) -185> 2014-11-08 18:15:52.056333 7fc5a6cfe700 5 -- op tracker -- seq: 47445, time: 2014-11-08 18:15:52.056167, event: throttled, op: MOSDECSubOpRead(4.bbs2 5781 ECSubRead(tid=25213, to_read={5c25a0bb/1000002b731.000000a6/head//4=0,4194304,b485a0bb/10000012c37.00000000/head//4=0,4194304,cea5a0bb/1000002b078.000001e0/head//4=0,4194304,4726a0bb/10000029591.00000016/head//4=0,4194304,2576a0bb/1000002a767.00000196/head//4=0,4194304}, attrs_to_read=)) -184> 2014-11-08 18:15:52.056339 7fc5a6cfe700 5 -- op tracker -- seq: 47445, time: 2014-11-08 18:15:52.056270, event: all_read, op: MOSDECSubOpRead(4.bbs2 5781 ECSubRead(tid=25213, to_read={5c25a0bb/1000002b731.000000a6/head//4=0,4194304,b485a0bb/10000012c37.00000000/head//4=0,4194304,cea5a0bb/1000002b078.000001e0/head//4=0,4194304,4726a0bb/10000029591.00000016/head//4=0,4194304,2576a0bb/1000002a767.00000196/head//4=0,4194304}, attrs_to_read=)) -183> 2014-11-08 18:15:52.056344 7fc5a6cfe700 5 -- op tracker -- seq: 47445, time: 0.000000, event: dispatched, op: MOSDECSubOpRead(4.bbs2 5781 ECSubRead(tid=25213, to_read={5c25a0bb/1000002b731.000000a6/head//4=0,4194304,b485a0bb/10000012c37.00000000/head//4=0,4194304,cea5a0bb/1000002b078.000001e0/head//4=0,4194304,4726a0bb/10000029591.00000016/head//4=0,4194304,2576a0bb/1000002a767.00000196/head//4=0,4194304}, attrs_to_read=)) -182> 2014-11-08 18:15:59.923084 7fc5b2b1b700 10 monclient: tick -181> 2014-11-08 18:15:59.923109 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:15:29.923108) -180> 2014-11-08 18:15:59.923121 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:15:59.923121; renew after: 2014-11-08 18:16:29.290049) -- no -179> 2014-11-08 18:16:00.757748 7fc5a5ff8700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.16:6804/1238 pipe(0x2244e5c0 sd=24 :6800 s=2 pgs=1755 cs=1 l=1 c=0x25996720).reader couldn't read tag, (0) Success -178> 2014-11-08 18:16:00.757788 7fc5a5ff8700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.16:6804/1238 pipe(0x2244e5c0 sd=24 :6800 s=2 pgs=1755 cs=1 l=1 c=0x25996720).fault (0) Success -177> 2014-11-08 18:16:00.758307 7fc5a6cfe700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6806/1411 pipe(0x4379080 sd=29 :33012 s=2 pgs=3008 cs=1 l=0 c=0x1c5c2680).reader couldn't read tag, (0) Success -176> 2014-11-08 18:16:00.758323 7fc5a6cfe700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6806/1411 pipe(0x4379080 sd=29 :33012 s=2 pgs=3008 cs=1 l=0 c=0x1c5c2680).fault (0) Success -175> 2014-11-08 18:16:00.758335 7fc5a6cfe700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6806/1411 pipe(0x4379080 sd=29 :33012 s=2 pgs=3008 cs=1 l=0 c=0x1c5c2680).fault with nothing to send, going to standby -174> 2014-11-08 18:16:00.758344 7fc5a2ec7700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.15:0/1411 pipe(0x437f380 sd=122 :6812 s=2 pgs=33 cs=1 l=1 c=0x1c9f8580).reader couldn't read tag, (0) Success -173> 2014-11-08 18:16:00.758368 7fc5a2ec7700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.15:0/1411 pipe(0x437f380 sd=122 :6812 s=2 pgs=33 cs=1 l=1 c=0x1c9f8580).fault (0) Success -172> 2014-11-08 18:16:00.758461 7fc5a3dd6700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/1550 pipe(0x1d22cfc0 sd=174 :6812 s=2 pgs=13516 cs=1 l=1 c=0x1eb9d6a0).reader couldn't read tag, (0) Success -171> 2014-11-08 18:16:00.758477 7fc5a3dd6700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/1550 pipe(0x1d22cfc0 sd=174 :6812 s=2 pgs=13516 cs=1 l=1 c=0x1eb9d6a0).fault (0) Success -170> 2014-11-08 18:16:00.758484 7fc5a2dc6700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.15:0/1411 pipe(0x437f900 sd=123 :6813 s=2 pgs=34 cs=1 l=1 c=0x1c8b7d20).reader couldn't read tag, (0) Success -169> 2014-11-08 18:16:00.758500 7fc5a2dc6700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.15:0/1411 pipe(0x437f900 sd=123 :6813 s=2 pgs=34 cs=1 l=1 c=0x1c8b7d20).fault (0) Success -168> 2014-11-08 18:16:00.758552 7fc5a50e9700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/1550 pipe(0x1d39d280 sd=33 :6813 s=2 pgs=13511 cs=1 l=1 c=0x1eb9c780).reader couldn't read tag, (0) Success -167> 2014-11-08 18:16:00.758565 7fc5a50e9700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/1550 pipe(0x1d39d280 sd=33 :6813 s=2 pgs=13511 cs=1 l=1 c=0x1eb9c780).fault (0) Success -166> 2014-11-08 18:16:00.759019 7fc5a56ef700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6806/12001550 pipe(0x22c1b700 sd=128 :6811 s=2 pgs=65553 cs=287 l=0 c=0x1e50aec0).reader couldn't read tag, (0) Success -165> 2014-11-08 18:16:00.759033 7fc5a56ef700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6806/12001550 pipe(0x22c1b700 sd=128 :6811 s=2 pgs=65553 cs=287 l=0 c=0x1e50aec0).fault (0) Success -164> 2014-11-08 18:16:00.759068 7fc5a56ef700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6806/12001550 pipe(0x22c1b700 sd=128 :6811 s=2 pgs=65553 cs=287 l=0 c=0x1e50aec0).fault with nothing to send, going to standby -163> 2014-11-08 18:16:00.760349 7fc5a0c26700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6801/1238 pipe(0x226c0580 sd=190 :6811 s=2 pgs=1029 cs=3 l=0 c=0x1cec3b20).reader couldn't read tag, (107) Transport endpoint is not connected -162> 2014-11-08 18:16:00.760371 7fc5a0c26700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6801/1238 pipe(0x226c0580 sd=190 :6811 s=2 pgs=1029 cs=3 l=0 c=0x1cec3b20).fault (107) Transport endpoint is not connected -161> 2014-11-08 18:16:00.760390 7fc5a0c26700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.16:6801/1238 pipe(0x226c0580 sd=190 :6811 s=2 pgs=1029 cs=3 l=0 c=0x1cec3b20).fault with nothing to send, going to standby -160> 2014-11-08 18:16:00.766974 7fc5a152e700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6811/1606 pipe(0x23e1bc80 sd=37 :53443 s=2 pgs=3797 cs=9 l=0 c=0x1c5c0580).reader couldn't read tag, (0) Success -159> 2014-11-08 18:16:00.766998 7fc5a152e700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6811/1606 pipe(0x23e1bc80 sd=37 :53443 s=2 pgs=3797 cs=9 l=0 c=0x1c5c0580).fault (0) Success -158> 2014-11-08 18:16:00.767012 7fc5a152e700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6811/1606 pipe(0x23e1bc80 sd=37 :53443 s=2 pgs=3797 cs=9 l=0 c=0x1c5c0580).fault with nothing to send, going to standby -157> 2014-11-08 18:16:00.767022 7fc5a61fa700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6802/1404 pipe(0x4379600 sd=43 :37320 s=2 pgs=7745 cs=7 l=0 c=0x1c7911e0).reader couldn't read tag, (0) Success -156> 2014-11-08 18:16:00.767032 7fc5a61fa700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6802/1404 pipe(0x4379600 sd=43 :37320 s=2 pgs=7745 cs=7 l=0 c=0x1c7911e0).fault (0) Success -155> 2014-11-08 18:16:00.767056 7fc5a61fa700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6802/1404 pipe(0x4379600 sd=43 :37320 s=2 pgs=7745 cs=7 l=0 c=0x1c7911e0).fault with nothing to send, going to standby -154> 2014-11-08 18:16:00.767783 7fc5a26bf700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.18:0/1404 pipe(0x1cd60580 sd=130 :6812 s=2 pgs=529 cs=1 l=1 c=0x1cce3180).reader couldn't read tag, (0) Success -153> 2014-11-08 18:16:00.767803 7fc5a26bf700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.18:0/1404 pipe(0x1cd60580 sd=130 :6812 s=2 pgs=529 cs=1 l=1 c=0x1cce3180).fault (0) Success -152> 2014-11-08 18:16:00.767936 7fc5a2ac3700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/31236 pipe(0x1d39b440 sd=158 :6812 s=2 pgs=10688 cs=1 l=1 c=0x1eb9d3e0).reader couldn't read tag, (0) Success -151> 2014-11-08 18:16:00.767949 7fc5a2ac3700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.16:0/31236 pipe(0x1d39b440 sd=158 :6812 s=2 pgs=10688 cs=1 l=1 c=0x1eb9d3e0).fault (0) Success -150> 2014-11-08 18:16:00.768005 7fc5a27c0700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/31236 pipe(0x215a7900 sd=157 :6813 s=2 pgs=10687 cs=1 l=1 c=0x1eb9b020).reader couldn't read tag, (0) Success -149> 2014-11-08 18:16:00.768017 7fc5a27c0700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.16:0/31236 pipe(0x215a7900 sd=157 :6813 s=2 pgs=10687 cs=1 l=1 c=0x1eb9b020).fault (0) Success -148> 2014-11-08 18:16:00.768543 7fc5a22bb700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.15:0/1606 pipe(0x1cd602c0 sd=133 :6812 s=2 pgs=38 cs=1 l=1 c=0x1cce2ec0).reader couldn't read tag, (0) Success -147> 2014-11-08 18:16:00.768557 7fc5a22bb700 2 -- 192.168.1.18:6812/4928 >> 192.168.1.15:0/1606 pipe(0x1cd602c0 sd=133 :6812 s=2 pgs=38 cs=1 l=1 c=0x1cce2ec0).fault (0) Success -146> 2014-11-08 18:16:00.769205 7fc5a21ba700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.15:0/1606 pipe(0x1cd60b00 sd=132 :6813 s=2 pgs=37 cs=1 l=1 c=0x1cce3440).reader couldn't read tag, (0) Success -145> 2014-11-08 18:16:00.769220 7fc5a21ba700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.15:0/1606 pipe(0x1cd60b00 sd=132 :6813 s=2 pgs=37 cs=1 l=1 c=0x1cce3440).fault (0) Success -144> 2014-11-08 18:16:00.772361 7fc5a25be700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.18:0/1404 pipe(0x1cd60dc0 sd=131 :6813 s=2 pgs=530 cs=1 l=1 c=0x1cce3700).reader couldn't read tag, (0) Success -143> 2014-11-08 18:16:00.772381 7fc5a25be700 2 -- 192.168.1.18:6813/4928 >> 192.168.1.18:0/1404 pipe(0x1cd60dc0 sd=131 :6813 s=2 pgs=530 cs=1 l=1 c=0x1cce3700).fault (0) Success -142> 2014-11-08 18:16:02.181518 7fc5a0f29700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.17:6809/32391 pipe(0x2244e880 sd=22 :6800 s=2 pgs=1435 cs=1 l=1 c=0x1dbe2c00).reader couldn't read tag, (0) Success -141> 2014-11-08 18:16:02.181555 7fc5a0f29700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.17:6809/32391 pipe(0x2244e880 sd=22 :6800 s=2 pgs=1435 cs=1 l=1 c=0x1dbe2c00).fault (0) Success -140> 2014-11-08 18:16:02.183715 7fc5cd36a700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.17:6806/32391 pipe(0x224ee880 sd=191 :6811 s=2 pgs=774 cs=5 l=0 c=0x1cec4620).reader couldn't read tag, (107) Transport endpoint is not connected -139> 2014-11-08 18:16:02.183733 7fc5cd36a700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.17:6806/32391 pipe(0x224ee880 sd=191 :6811 s=2 pgs=774 cs=5 l=0 c=0x1cec4620).fault (107) Transport endpoint is not connected -138> 2014-11-08 18:16:02.183747 7fc5cd36a700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.17:6806/32391 pipe(0x224ee880 sd=191 :6811 s=2 pgs=774 cs=5 l=0 c=0x1cec4620).fault with nothing to send, going to standby -137> 2014-11-08 18:16:02.269785 7fc5a5bf4700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.15:6804/1266 pipe(0x215a4780 sd=52 :6800 s=2 pgs=112 cs=1 l=1 c=0x25994fc0).reader couldn't read tag, (0) Success -136> 2014-11-08 18:16:02.269825 7fc5a5bf4700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.15:6804/1266 pipe(0x215a4780 sd=52 :6800 s=2 pgs=112 cs=1 l=1 c=0x25994fc0).fault (0) Success -135> 2014-11-08 18:16:02.270252 7fc5c746a700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6801/1266 pipe(0x2342f900 sd=194 :6811 s=2 pgs=64 cs=5 l=0 c=0x1cec4780).reader couldn't read tag, (107) Transport endpoint is not connected -134> 2014-11-08 18:16:02.270270 7fc5c746a700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6801/1266 pipe(0x2342f900 sd=194 :6811 s=2 pgs=64 cs=5 l=0 c=0x1cec4780).fault (107) Transport endpoint is not connected -133> 2014-11-08 18:16:02.270284 7fc5c746a700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.15:6801/1266 pipe(0x2342f900 sd=194 :6811 s=2 pgs=64 cs=5 l=0 c=0x1cec4780).fault with nothing to send, going to standby -132> 2014-11-08 18:16:02.275699 7fc5a1b34700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.18:6810/3588 pipe(0x215a6880 sd=196 :6800 s=2 pgs=409 cs=1 l=1 c=0x25994360).reader couldn't read tag, (0) Success -131> 2014-11-08 18:16:02.275729 7fc5a1b34700 2 -- 192.168.1.18:6800/4928 >> 192.168.1.18:6810/3588 pipe(0x215a6880 sd=196 :6800 s=2 pgs=409 cs=1 l=1 c=0x25994360).fault (0) Success -130> 2014-11-08 18:16:02.275806 7fc5a0d27700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6807/3588 pipe(0x226c02c0 sd=184 :6811 s=2 pgs=208 cs=3 l=0 c=0x1cec4200).reader couldn't read tag, (107) Transport endpoint is not connected -129> 2014-11-08 18:16:02.275820 7fc5a0d27700 2 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6807/3588 pipe(0x226c02c0 sd=184 :6811 s=2 pgs=208 cs=3 l=0 c=0x1cec4200).fault (107) Transport endpoint is not connected -128> 2014-11-08 18:16:02.275838 7fc5a0d27700 0 -- 192.168.1.18:6811/4928 >> 192.168.1.18:6807/3588 pipe(0x226c02c0 sd=184 :6811 s=2 pgs=208 cs=3 l=0 c=0x1cec4200).fault with nothing to send, going to standby -127> 2014-11-08 18:16:02.985549 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -126> 2014-11-08 18:16:07.985682 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -125> 2014-11-08 18:16:09.923198 7fc5b2b1b700 10 monclient: tick -124> 2014-11-08 18:16:09.923225 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:15:39.923223) -123> 2014-11-08 18:16:09.923238 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:16:09.923238; renew after: 2014-11-08 18:16:29.290049) -- no -122> 2014-11-08 18:16:12.985790 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -121> 2014-11-08 18:16:12.985817 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -120> 2014-11-08 18:16:17.985897 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -119> 2014-11-08 18:16:17.985918 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -118> 2014-11-08 18:16:19.923319 7fc5b2b1b700 10 monclient: tick -117> 2014-11-08 18:16:19.923347 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:15:49.923345) -116> 2014-11-08 18:16:19.923359 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:16:19.923359; renew after: 2014-11-08 18:16:29.290049) -- no -115> 2014-11-08 18:16:22.985984 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -114> 2014-11-08 18:16:22.986006 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -113> 2014-11-08 18:16:27.986083 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -112> 2014-11-08 18:16:27.986099 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -111> 2014-11-08 18:16:29.925276 7fc5b2b1b700 10 monclient: tick -110> 2014-11-08 18:16:29.925321 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:15:59.925317) -109> 2014-11-08 18:16:29.925342 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:16:29.925341; renew after: 2014-11-08 18:16:29.290049) -- yes -108> 2014-11-08 18:16:29.925346 7fc5b2b1b700 10 monclient: renew_subs -107> 2014-11-08 18:16:29.925353 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -106> 2014-11-08 18:16:29.925369 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x22e9d460 con 0x44ed6a0 -105> 2014-11-08 18:16:32.986340 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -104> 2014-11-08 18:16:32.986359 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -103> 2014-11-08 18:16:37.986429 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -102> 2014-11-08 18:16:37.986445 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -101> 2014-11-08 18:16:39.925480 7fc5b2b1b700 10 monclient: tick -100> 2014-11-08 18:16:39.925502 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:16:09.925501) -99> 2014-11-08 18:16:39.925518 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:16:39.925517; renew after: 2014-11-08 18:16:29.290049) -- yes -98> 2014-11-08 18:16:39.925526 7fc5b2b1b700 10 monclient: renew_subs -97> 2014-11-08 18:16:39.925536 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -96> 2014-11-08 18:16:39.925543 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x22e994a0 con 0x44ed6a0 -95> 2014-11-08 18:16:42.986521 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -94> 2014-11-08 18:16:42.986551 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -93> 2014-11-08 18:16:47.986631 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -92> 2014-11-08 18:16:47.986644 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -91> 2014-11-08 18:16:49.925627 7fc5b2b1b700 10 monclient: tick -90> 2014-11-08 18:16:49.925647 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:16:19.925645) -89> 2014-11-08 18:16:49.925658 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:16:49.925658; renew after: 2014-11-08 18:16:29.290049) -- yes -88> 2014-11-08 18:16:49.925667 7fc5b2b1b700 10 monclient: renew_subs -87> 2014-11-08 18:16:49.925676 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -86> 2014-11-08 18:16:49.925696 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x22e9dbe0 con 0x44ed6a0 -85> 2014-11-08 18:16:52.986716 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -84> 2014-11-08 18:16:52.986740 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -83> 2014-11-08 18:16:57.986812 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -82> 2014-11-08 18:16:57.986833 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -81> 2014-11-08 18:16:59.926010 7fc5b2b1b700 10 monclient: tick -80> 2014-11-08 18:16:59.926038 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:16:29.926037) -79> 2014-11-08 18:16:59.926052 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:16:59.926051; renew after: 2014-11-08 18:16:29.290049) -- yes -78> 2014-11-08 18:16:59.926055 7fc5b2b1b700 10 monclient: renew_subs -77> 2014-11-08 18:16:59.926059 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -76> 2014-11-08 18:16:59.926069 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x22e99e00 con 0x44ed6a0 -75> 2014-11-08 18:17:02.986915 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -74> 2014-11-08 18:17:02.986939 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -73> 2014-11-08 18:17:07.987010 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -72> 2014-11-08 18:17:07.987045 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -71> 2014-11-08 18:17:09.926155 7fc5b2b1b700 10 monclient: tick -70> 2014-11-08 18:17:09.926181 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:16:39.926179) -69> 2014-11-08 18:17:09.926198 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:17:09.926198; renew after: 2014-11-08 18:16:29.290049) -- yes -68> 2014-11-08 18:17:09.926211 7fc5b2b1b700 10 monclient: renew_subs -67> 2014-11-08 18:17:09.926215 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -66> 2014-11-08 18:17:09.926225 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x22e9eae0 con 0x44ed6a0 -65> 2014-11-08 18:17:12.987117 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -64> 2014-11-08 18:17:12.987134 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -63> 2014-11-08 18:17:17.987205 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -62> 2014-11-08 18:17:17.987233 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -61> 2014-11-08 18:17:19.926312 7fc5b2b1b700 10 monclient: tick -60> 2014-11-08 18:17:19.926350 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:16:49.926348) -59> 2014-11-08 18:17:19.926362 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:17:19.926361; renew after: 2014-11-08 18:16:29.290049) -- yes -58> 2014-11-08 18:17:19.926365 7fc5b2b1b700 10 monclient: renew_subs -57> 2014-11-08 18:17:19.926369 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -56> 2014-11-08 18:17:19.926377 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x22e9fda0 con 0x44ed6a0 -55> 2014-11-08 18:17:22.987303 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -54> 2014-11-08 18:17:22.987323 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -53> 2014-11-08 18:17:27.987395 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -52> 2014-11-08 18:17:27.987420 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -51> 2014-11-08 18:17:29.926465 7fc5b2b1b700 10 monclient: tick -50> 2014-11-08 18:17:29.926490 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:16:59.926489) -49> 2014-11-08 18:17:29.926503 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:17:29.926503; renew after: 2014-11-08 18:16:29.290049) -- yes -48> 2014-11-08 18:17:29.926508 7fc5b2b1b700 10 monclient: renew_subs -47> 2014-11-08 18:17:29.926517 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -46> 2014-11-08 18:17:29.926524 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x227e7260 con 0x44ed6a0 -45> 2014-11-08 18:17:32.987496 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -44> 2014-11-08 18:17:32.987511 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -43> 2014-11-08 18:17:37.987585 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -42> 2014-11-08 18:17:37.987615 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -41> 2014-11-08 18:17:39.926611 7fc5b2b1b700 10 monclient: tick -40> 2014-11-08 18:17:39.926634 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:17:09.926632) -39> 2014-11-08 18:17:39.926646 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:17:39.926646; renew after: 2014-11-08 18:16:29.290049) -- yes -38> 2014-11-08 18:17:39.926655 7fc5b2b1b700 10 monclient: renew_subs -37> 2014-11-08 18:17:39.926659 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -36> 2014-11-08 18:17:39.926672 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x227e6ea0 con 0x44ed6a0 -35> 2014-11-08 18:17:42.987684 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -34> 2014-11-08 18:17:42.987707 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -33> 2014-11-08 18:17:47.987784 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -32> 2014-11-08 18:17:47.987832 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -31> 2014-11-08 18:17:49.926762 7fc5b2b1b700 10 monclient: tick -30> 2014-11-08 18:17:49.926781 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:17:19.926780) -29> 2014-11-08 18:17:49.926790 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:17:49.926790; renew after: 2014-11-08 18:16:29.290049) -- yes -28> 2014-11-08 18:17:49.926794 7fc5b2b1b700 10 monclient: renew_subs -27> 2014-11-08 18:17:49.926801 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -26> 2014-11-08 18:17:49.926807 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x45341a0 con 0x44ed6a0 -25> 2014-11-08 18:17:52.987905 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -24> 2014-11-08 18:17:52.987924 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -23> 2014-11-08 18:17:57.987996 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -22> 2014-11-08 18:17:57.988018 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -21> 2014-11-08 18:17:59.926891 7fc5b2b1b700 10 monclient: tick -20> 2014-11-08 18:17:59.926918 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:17:29.926917) -19> 2014-11-08 18:17:59.926936 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:17:59.926935; renew after: 2014-11-08 18:16:29.290049) -- yes -18> 2014-11-08 18:17:59.926953 7fc5b2b1b700 10 monclient: renew_subs -17> 2014-11-08 18:17:59.926958 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -16> 2014-11-08 18:17:59.926970 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x4533a20 con 0x44ed6a0 -15> 2014-11-08 18:18:02.988091 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -14> 2014-11-08 18:18:02.988109 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -13> 2014-11-08 18:18:07.988249 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -12> 2014-11-08 18:18:07.988270 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -11> 2014-11-08 18:18:09.927074 7fc5b2b1b700 10 monclient: tick -10> 2014-11-08 18:18:09.927097 7fc5b2b1b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-11-08 18:17:39.927095) -9> 2014-11-08 18:18:09.927108 7fc5b2b1b700 10 monclient: renew subs? (now: 2014-11-08 18:18:09.927108; renew after: 2014-11-08 18:16:29.290049) -- yes -8> 2014-11-08 18:18:09.927114 7fc5b2b1b700 10 monclient: renew_subs -7> 2014-11-08 18:18:09.927122 7fc5b2b1b700 10 monclient: _send_mon_message to mon.ceph-04 at 192.168.1.18:6789/0 -6> 2014-11-08 18:18:09.927131 7fc5b2b1b700 1 -- 192.168.1.18:6800/4928 --> 192.168.1.18:6789/0 -- mon_subscribe({monmap=6+,osd_pg_creates=0}) v2 -- ?+0 0x4535640 con 0x44ed6a0 -5> 2014-11-08 18:18:12.988344 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -4> 2014-11-08 18:18:12.988368 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -3> 2014-11-08 18:18:17.988440 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::recovery_tp thread 0x7fc5abb0d700' had timed out after 30 -2> 2014-11-08 18:18:17.988466 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had timed out after 15 -1> 2014-11-08 18:18:17.988468 7fc5c9375700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fc5aeb13700' had suicide timed out after 150 0> 2014-11-08 18:18:18.249885 7fc5c9375700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fc5c9375700 time 2014-11-08 18:18:18.109404 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xab6905] 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0x9ef1a9] 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0x9efa9e] 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x9f01bc] 5: (CephContextServiceThread::entry()+0x15b) [0xacb3fb] 6: (()+0x7ee5) [0x7fc5cc365ee5] 7: (clone()+0x6d) [0x7fc5cb04bb8d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.9.log --- end dump of recent events --- 2014-11-08 18:18:18.417098 7fc5c9375700 -1 *** Caught signal (Aborted) ** in thread 7fc5c9375700 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: /usr/bin/ceph-osd() [0x9c4132] 2: (()+0xf6d0) [0x7fc5cc36d6d0] 3: (gsignal()+0x37) [0x7fc5caf8c877] 4: (abort()+0x148) [0x7fc5caf8df68] 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fc5cb892dd5] 6: (()+0x5ed46) [0x7fc5cb890d46] 7: (()+0x5ed73) [0x7fc5cb890d73] 8: (()+0x5ef93) [0x7fc5cb890f93] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xab6afa] 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0x9ef1a9] 11: (ceph::HeartbeatMap::is_healthy()+0xde) [0x9efa9e] 12: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x9f01bc] 13: (CephContextServiceThread::entry()+0x15b) [0xacb3fb] 14: (()+0x7ee5) [0x7fc5cc365ee5] 15: (clone()+0x6d) [0x7fc5cb04bb8d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2014-11-08 18:18:18.417098 7fc5c9375700 -1 *** Caught signal (Aborted) ** in thread 7fc5c9375700 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: /usr/bin/ceph-osd() [0x9c4132] 2: (()+0xf6d0) [0x7fc5cc36d6d0] 3: (gsignal()+0x37) [0x7fc5caf8c877] 4: (abort()+0x148) [0x7fc5caf8df68] 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fc5cb892dd5] 6: (()+0x5ed46) [0x7fc5cb890d46] 7: (()+0x5ed73) [0x7fc5cb890d73] 8: (()+0x5ef93) [0x7fc5cb890f93] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xab6afa] 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0x9ef1a9] 11: (ceph::HeartbeatMap::is_healthy()+0xde) [0x9efa9e] 12: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x9f01bc] 13: (CephContextServiceThread::entry()+0x15b) [0xacb3fb] 14: (()+0x7ee5) [0x7fc5cc365ee5] 15: (clone()+0x6d) [0x7fc5cb04bb8d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.9.log --- end dump of recent events --- 2014-11-08 20:05:45.761554 7f8332324800 0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 15665 2014-11-08 20:05:45.817011 7f8332324800 0 filestore(/var/lib/ceph/osd/ceph-9) backend btrfs (magic 0x9123683e) 2014-11-08 20:05:46.016508 7f8332324800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: FIEMAP ioctl is supported and appears to work 2014-11-08 20:05:46.016585 7f8332324800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-11-08 20:05:46.498289 7f8332324800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-11-08 20:05:46.498445 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: CLONE_RANGE ioctl is supported 2014-11-08 20:05:47.481557 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: SNAP_CREATE is supported 2014-11-08 20:05:47.482035 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: SNAP_DESTROY is supported 2014-11-08 20:05:47.485604 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: START_SYNC is supported (transid 68985) 2014-11-08 20:05:47.848164 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: WAIT_SYNC is supported 2014-11-08 20:05:47.898914 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: SNAP_CREATE_V2 is supported 2014-11-08 20:05:54.274083 7f8332324800 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling PARALLEL journal mode: fs, checkpoint is enabled 2014-11-08 20:05:55.110486 7f8332324800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-11-08 20:05:55.110550 7f8332324800 1 journal _open /var/lib/ceph/journal/ceph-9/journal fd 20: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-11-08 20:06:06.223972 7f8332324800 1 journal _open /var/lib/ceph/journal/ceph-9/journal fd 20: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-11-08 20:06:08.203137 7f8332324800 1 journal close /var/lib/ceph/journal/ceph-9/journal 2014-11-08 20:06:08.574813 7f8332324800 0 filestore(/var/lib/ceph/osd/ceph-9) backend btrfs (magic 0x9123683e) 2014-11-08 20:06:09.018068 7f8332324800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: FIEMAP ioctl is supported and appears to work 2014-11-08 20:06:09.018105 7f8332324800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-11-08 20:06:10.222235 7f8332324800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-11-08 20:06:10.222368 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: CLONE_RANGE ioctl is supported 2014-11-08 20:06:10.654656 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: SNAP_CREATE is supported 2014-11-08 20:06:10.655074 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: SNAP_DESTROY is supported 2014-11-08 20:06:10.793089 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: START_SYNC is supported (transid 68993) 2014-11-08 20:06:11.201303 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: WAIT_SYNC is supported 2014-11-08 20:06:11.220047 7f8332324800 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: SNAP_CREATE_V2 is supported 2014-11-08 20:06:12.456033 7f8332324800 0 filestore(/var/lib/ceph/osd/ceph-9) mount: PARALLEL journal mode explicitly enabled in conf 2014-11-08 20:06:12.458110 7f8332324800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-11-08 20:06:12.458114 7f8332324800 1 journal _open /var/lib/ceph/journal/ceph-9/journal fd 20: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-11-08 20:06:12.461519 7f8332324800 1 journal _open /var/lib/ceph/journal/ceph-9/journal fd 20: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0 2014-11-08 20:06:12.554856 7f8332324800 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2014-11-08 20:06:12.620030 7f8332324800 0 osd.9 5776 crush map has features 2303210029056, adjusting msgr requires for clients 2014-11-08 20:06:12.620043 7f8332324800 0 osd.9 5776 crush map has features 2578087936000 was 8705, adjusting msgr requires for mons 2014-11-08 20:06:12.620047 7f8332324800 0 osd.9 5776 crush map has features 2578087936000, adjusting msgr requires for osds 2014-11-08 20:06:12.620062 7f8332324800 0 osd.9 5776 load_pgs 2014-11-08 20:06:14.220854 7f8326ecb700 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) destroy_checkpoint: ioctl SNAP_DESTROY got (2) No such file or directory 2014-11-08 20:06:14.220894 7f8326ecb700 -1 filestore(/var/lib/ceph/osd/ceph-9) unable to destroy snap 'snap_2098628' got (2) No such file or directory 2014-11-08 20:06:15.404613 7f8326ecb700 0 btrfsfilestorebackend(/var/lib/ceph/osd/ceph-9) destroy_checkpoint: ioctl SNAP_DESTROY got (2) No such file or directory 2014-11-08 20:06:15.404643 7f8326ecb700 -1 filestore(/var/lib/ceph/osd/ceph-9) unable to destroy snap 'snap_2098629' got (2) No such file or directory 2014-11-08 20:06:34.160158 7f8332324800 0 osd.9 5776 load_pgs opened 107 pgs 2014-11-08 20:06:34.170059 7f8332324800 -1 osd.9 5776 set_disk_tp_priority(22) Invalid argument: osd_disk_thread_ioprio_class is but only the following values are allowed: idle, be or rt 2014-11-08 20:06:34.189563 7f8320386700 0 osd.9 5776 ignoring osdmap until we have initialized 2014-11-08 20:06:34.189794 7f8320386700 0 osd.9 5776 ignoring osdmap until we have initialized 2014-11-08 20:06:34.823487 7f8332324800 0 osd.9 5776 done with init, starting boot process 2014-11-08 20:06:35.128548 7f8309739700 0 -- 192.168.1.18:6811/15665 >> 192.168.1.16:6815/6031236 pipe(0x3f9e5c0 sd=102 :6811 s=0 pgs=0 cs=0 l=0 c=0x1c67f4e0).accept connect_seq 0 vs existing 0 state connecting 2014-11-08 20:06:36.504626 7f8307618700 0 -- 192.168.1.18:6811/15665 >> 192.168.1.15:6801/1266 pipe(0x1ca11b80 sd=114 :6811 s=0 pgs=0 cs=0 l=0 c=0x1c67f640).accept connect_seq 0 vs existing 0 state connecting 2014-11-08 20:06:36.504985 7f8307416700 0 -- 192.168.1.18:6811/15665 >> 192.168.1.18:6807/3588 pipe(0x1ca123c0 sd=117 :6811 s=0 pgs=0 cs=0 l=0 c=0x1cb40b00).accept connect_seq 0 vs existing 0 state connecting 2014-11-08 20:06:36.505298 7f8307315700 0 -- 192.168.1.18:6811/15665 >> 192.168.1.17:6806/32391 pipe(0x1ca12100 sd=118 :6811 s=0 pgs=0 cs=0 l=0 c=0x1cb40840).accept connect_seq 0 vs existing 0 state connecting 2014-11-08 20:06:36.505519 7f8307517700 0 -- 192.168.1.18:6811/15665 >> 192.168.1.16:6801/1238 pipe(0x1ca118c0 sd=116 :6811 s=0 pgs=0 cs=0 l=0 c=0x1cb40420).accept connect_seq 0 vs existing 0 state connecting
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com