On Fri, Apr 10, 2015 at 09:55:20AM +0100, Jacob Reid wrote:
> On Thu, Apr 09, 2015 at 05:21:47PM +0100, Jacob Reid wrote:
> > On Thu, Apr 09, 2015 at 08:46:07AM -0700, Gregory Farnum wrote:
> > > On Thu, Apr 9, 2015 at 8:14 AM, Jacob Reid <lists-ceph@xxxxxxxxxxxxxxxx> wrote:
> > > > On Thu, Apr 09, 2015 at 06:43:45AM -0700, Gregory Farnum wrote:
> > > >> You can turn up debugging ("debug osd = 10" and "debug filestore = 10"
> > > >> are probably enough, or maybe 20 each) and see what comes out to get
> > > >> more information about why the threads are stuck.
> > > >>
> > > >> But just from the log my answer is the same as before, and now I don't
> > > >> trust that controller (or maybe its disks), regardless of what it's
> > > >> admitting to. ;)
> > > >> -Greg
> > > >>
> > > >
> > > > Ran with osd and filestore debug both at 20; still nothing jumping out at me. Logfile attached as it got huge fairly quickly, but mostly seems to be the same extra lines. I tried running some test I/O on the drives in question to try and provoke some kind of problem, but they seem fine now...
> > >
> > > Okay, this is strange. Something very wonky is happening with your
> > > scheduler — it looks like these threads are all idle, and they're
> > > scheduling wakeups that handle an appreciable amount of time after
> > > they're supposed to. For instance:
> > > 2015-04-09 15:56:55.953116 7f70a7963700 20
> > > filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.416704
> > > 2015-04-09 15:56:55.953153 7f70a7963700 20
> > > filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for
> > > max_interval 5.000000
> > >
> > > This is the thread that syncs your backing store, and it always sets
> > > itself to get woken up at 5-second intervals — but here it took >5.4
> > > seconds, and later on in your log it takes more than 6 seconds.
> > > It looks like all the threads which are getting timed out are also
> > > idle, but are taking so much longer to wake up than they're set for
> > > that they get a timeout warning.
> > >
> > > There might be some bugs in here where we're expecting wakeups to be
> > > more precise than they can be, but these sorts of misses are
> > > definitely not normal. Is this server overloaded on the CPU? Have you
> > > done something to make the scheduler or wakeups wonky?
> > > -Greg
> >
> > CPU load is minimal - the host does nothing but run OSDs and has 8 cores that are all sitting idle with a load average of 0.1. I haven't done anything to scheduling. That was with the debug logging on, if that could be the cause of any delays. A scheduler issue seems possible - I haven't done anything to it, but `time sleep 5` run a few times returns anything spread randomly from 5.002 to 7.1(!) seconds but mostly in the 5.5-6.0 region where it managed fairly consistently <5.2 on the other servers in the cluster and <5.02 on my desktop. I have disabled the CPU power saving mode as the only thing I could think of that might be having an effect on this, and running the same test again gives more sane results... we'll see if this reflects in the OSD logs or not, I guess. If this is the cause, it's probably something that the next version might want to make a specific warning case of detecting. I will keep you updated as to their behaviour now...
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> Overnight, nothing changed - I am no longer seeing the timeout in the logs but all the OSDs in questions are still happily sitting at booting and showing as down in the tree. Debug 20 logfile attached again.
...and here actually *is* the logfile, which I managed to forget... must be Friday, I guess.
2015-04-10 09:05:21.880199 7fd4769f5700 -1 osd.15 3814 *** Got signal Terminated ***
2015-04-10 09:05:21.880280 7fd4769f5700 0 osd.15 3814 prepare_to_stop starting shutdown
2015-04-10 09:05:21.880293 7fd4769f5700 -1 osd.15 3814 shutdown
2015-04-10 09:05:21.880894 7fd4769f5700 10 osd.15 3814 recovery tp stopped
2015-04-10 09:05:21.881055 7fd4769f5700 10 osd.15 3814 osd tp stopped
2015-04-10 09:05:21.881189 7fd4769f5700 10 osd.15 3814 command tp stopped
2015-04-10 09:05:21.881283 7fd4769f5700 10 osd.15 3814 disk tp paused (new)
2015-04-10 09:05:21.881309 7fd4769f5700 10 osd.15 3814 stopping agent
2015-04-10 09:05:21.881334 7fd4772f7700 10 osd.15 3814 agent_entry finish
2015-04-10 09:05:21.881388 7fd4769f5700 10 osd.15 3814 reset_heartbeat_peers
2015-04-10 09:05:21.881546 7fd4769f5700 10 osd.15 3814 noting clean unmount in epoch 3814
2015-04-10 09:05:21.881571 7fd4769f5700 10 osd.15 3814 write_superblock sb(71783e7e-b856-4b96-a478-27191a56bf17 osd.15 e411ce75-c0f6-47f6-b58f-804a093fae81 e3814 [2527,3814] lci=[0,3814])
2015-04-10 09:05:21.881632 7fd4769f5700 5 filestore(/var/lib/ceph/osd/osd.15) queue_transactions new osr(default 0x234ab98)/0x234ab98
2015-04-10 09:05:21.881651 7fd4769f5700 10 journal op_submit_start 30
2015-04-10 09:05:21.881654 7fd4769f5700 5 filestore(/var/lib/ceph/osd/osd.15) queue_transactions (writeahead) 30 0x7fd4769f48f0
2015-04-10 09:05:21.881659 7fd4769f5700 10 journal op_journal_transactions 30 0x7fd4769f48f0
2015-04-10 09:05:21.881684 7fd4769f5700 5 journal submit_entry seq 30 len 492 (0x246ce10)
2015-04-10 09:05:21.881687 7fd4769f5700 30 journal XXX throttle take 492
2015-04-10 09:05:21.881709 7fd4769f5700 10 journal op_submit_finish 30
2015-04-10 09:05:21.881750 7fd488a62700 20 journal write_thread_entry woke up
2015-04-10 09:05:21.881759 7fd488a62700 10 journal room 15999168511 max_size 15999172608 pos 44740608 header.start 44740608 top 4096
2015-04-10 09:05:21.881763 7fd488a62700 10 journal check_for_full at 44740608 : 4096 < 15999168511
2015-04-10 09:05:21.881766 7fd488a62700 15 journal prepare_single_write 1 will write 44740608 : seq 30 len 492 -> 4096 (head 40 pre_pad 0 ebl 492 post_pad 3524 tail 40) (ebl alignment -1)
2015-04-10 09:05:21.881783 7fd488a62700 20 journal prepare_multi_write queue_pos now 44744704
2015-04-10 09:05:21.881786 7fd488a62700 15 journal do_aio_write writing 44740608~4096
2015-04-10 09:05:21.881811 7fd488a62700 20 journal write_aio_bl 44740608~4096 seq 30
2015-04-10 09:05:21.881817 7fd488a62700 20 journal write_aio_bl .. 44740608~4096 in 1
2015-04-10 09:05:21.881886 7fd488a62700 5 journal put_throttle finished 1 ops and 492 bytes, now 0 ops and 0 bytes
2015-04-10 09:05:21.881894 7fd488a62700 20 journal write_thread_entry going to sleep
2015-04-10 09:05:21.881927 7fd488261700 20 journal write_finish_thread_entry waiting for aio(s)
2015-04-10 09:05:21.881972 7fd488261700 10 journal write_finish_thread_entry aio 44740608~4096 done
2015-04-10 09:05:21.881978 7fd488261700 20 journal check_aio_completion
2015-04-10 09:05:21.881980 7fd488261700 20 journal check_aio_completion completed seq 30 44740608~4096
2015-04-10 09:05:21.881985 7fd488261700 20 journal check_aio_completion queueing finishers through seq 30
2015-04-10 09:05:21.881990 7fd488261700 10 journal queue_completions_thru seq 30 queueing seq 30 0x246ce10 lat 0.000297
2015-04-10 09:05:21.882004 7fd488261700 20 journal write_finish_thread_entry sleeping
2015-04-10 09:05:21.882040 7fd487a60700 5 filestore(/var/lib/ceph/osd/osd.15) _journaled_ahead 0x24a1000 seq 30 osr(default 0x234ab98) 0x7fd4769f48f0
2015-04-10 09:05:21.882051 7fd487a60700 5 filestore(/var/lib/ceph/osd/osd.15) queue_op 0x24a1000 seq 30 osr(default 0x234ab98) 486 bytes (queue has 1 ops and 486 bytes)
2015-04-10 09:05:21.882082 7fd48725f700 10 journal op_apply_start 30 open_ops 0 -> 1
2015-04-10 09:05:21.882089 7fd48725f700 5 filestore(/var/lib/ceph/osd/osd.15) _do_op 0x24a1000 seq 30 osr(default 0x234ab98)/0x234ab98 start
2015-04-10 09:05:21.882107 7fd48725f700 10 filestore(/var/lib/ceph/osd/osd.15) _do_transaction on 0x7fd4769f48f0
2015-04-10 09:05:21.882132 7fd48725f700 15 filestore(/var/lib/ceph/osd/osd.15) write meta/23c2fcde/osd_superblock/0//-1 0~358
2015-04-10 09:05:21.882230 7fd48725f700 10 filestore(/var/lib/ceph/osd/osd.15) write meta/23c2fcde/osd_superblock/0//-1 0~358 = 358
2015-04-10 09:05:21.882242 7fd48725f700 10 journal op_apply_finish 30 open_ops 1 -> 0, max_applied_seq 29 -> 30
2015-04-10 09:05:21.882245 7fd48725f700 10 filestore(/var/lib/ceph/osd/osd.15) _do_op 0x24a1000 seq 30 r = 0, finisher 0x23819c0 0
2015-04-10 09:05:21.882255 7fd48725f700 10 filestore(/var/lib/ceph/osd/osd.15) _finish_op 0x24a1000 seq 30 osr(default 0x234ab98)/0x234ab98
2015-04-10 09:05:21.882353 7fd4769f5700 10 osd.15 3814 syncing store
2015-04-10 09:05:21.882370 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) flush
2015-04-10 09:05:21.882374 7fd4769f5700 10 journal waiting for completions to empty
2015-04-10 09:05:21.882378 7fd4769f5700 10 journal flush waiting for finisher
2015-04-10 09:05:21.882380 7fd4769f5700 10 journal flush done
2015-04-10 09:05:21.882394 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) flush draining ondisk finisher
2015-04-10 09:05:21.882397 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) _flush_op_queue draining op tp
2015-04-10 09:05:21.882400 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) _flush_op_queue waiting for apply finisher
2015-04-10 09:05:21.882401 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) flush complete
2015-04-10 09:05:21.882416 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) start_sync
2015-04-10 09:05:21.882420 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) sync waiting
2015-04-10 09:05:21.882427 7fd489263700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 4.391464
2015-04-10 09:05:21.882442 7fd489263700 10 journal commit_start max_applied_seq 30, open_ops 0
2015-04-10 09:05:21.882446 7fd489263700 10 journal commit_start blocked, all open_ops have completed
2015-04-10 09:05:21.882448 7fd489263700 10 journal commit_start committing 30, still blocked
2015-04-10 09:05:21.882510 7fd489263700 10 journal commit_start
2015-04-10 09:05:21.882523 7fd489263700 15 filestore(/var/lib/ceph/osd/osd.15) sync_entry committing 30 sync_epoch 1
2015-04-10 09:05:21.882531 7fd489263700 10 journal commit_started committing 30, unblocking
2015-04-10 09:05:21.882551 7fd489263700 15 genericfilestorebackend(/var/lib/ceph/osd/osd.15) syncfs: doing a full sync (syncfs(2) if possible)
2015-04-10 09:05:21.884945 7fd489263700 10 filestore(/var/lib/ceph/osd/osd.15) sync_entry commit took 0.002428, interval was 4.393980
2015-04-10 09:05:21.884969 7fd489263700 10 journal commit_finish thru 30
2015-04-10 09:05:21.884977 7fd489263700 5 journal committed_thru 30 (last_committed_seq 29)
2015-04-10 09:05:21.884989 7fd489263700 10 journal header: block_size 4096 alignment 4096 max_size 15999172608
2015-04-10 09:05:21.884992 7fd489263700 10 journal header: start 44744704
2015-04-10 09:05:21.884994 7fd489263700 10 journal write_pos 44744704
2015-04-10 09:05:21.884996 7fd489263700 10 journal committed_thru done
2015-04-10 09:05:21.885014 7fd489263700 15 filestore(/var/lib/ceph/osd/osd.15) sync_entry committed to op_seq 30
2015-04-10 09:05:21.885037 7fd489263700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:21.885131 7fd4769f5700 10 filestore(/var/lib/ceph/osd/osd.15) sync done
2015-04-10 09:05:21.885145 7fd4769f5700 5 filestore(/var/lib/ceph/osd/osd.15) umount /var/lib/ceph/osd/osd.15
2015-04-10 09:05:21.885173 7fd489263700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry force_sync set
2015-04-10 09:05:21.885185 7fd489263700 10 journal commit_start max_applied_seq 30, open_ops 0
2015-04-10 09:05:21.885188 7fd489263700 10 journal commit_start blocked, all open_ops have completed
2015-04-10 09:05:21.885190 7fd489263700 10 journal commit_start nothing to do
2015-04-10 09:05:21.885192 7fd489263700 10 journal commit_start
2015-04-10 09:05:21.885495 7fd4769f5700 10 journal journal_stop
2015-04-10 09:05:21.885573 7fd4769f5700 1 journal close /dev/sdj4
2015-04-10 09:05:21.885595 7fd488a62700 20 journal write_thread_entry woke up
2015-04-10 09:05:21.885603 7fd488a62700 20 journal prepare_multi_write queue_pos now 44744704
2015-04-10 09:05:21.885610 7fd488261700 10 journal write_finish_thread_entry exit
2015-04-10 09:05:21.885630 7fd488a62700 15 journal do_write writing 44744704~0 + header
2015-04-10 09:05:21.885782 7fd488a62700 20 journal do_write latency 0.000148
2015-04-10 09:05:21.885795 7fd488a62700 20 journal do_write queueing finishers through seq 30
2015-04-10 09:05:21.885800 7fd488a62700 5 journal put_throttle finished 0 ops and 0 bytes, now 0 ops and 0 bytes
2015-04-10 09:05:21.885806 7fd488a62700 10 journal write_thread_entry finish
2015-04-10 09:05:21.886622 7fd4769f5700 10 osd.15 3814 Store synced
2015-04-10 09:05:21.886988 7fd4769f5700 1 -- 192.168.6.152:6805/3910 mark_down 0x24779a0 -- 0x2373c80
2015-04-10 09:05:21.887011 7fd4769f5700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=2 pgs=2397361 cs=1 l=1 c=0x24779a0).unregister_pipe
2015-04-10 09:05:21.887029 7fd4769f5700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=2 pgs=2397361 cs=1 l=1 c=0x24779a0).stop
2015-04-10 09:05:21.887100 7fd476af6700 2 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).reader couldn't read tag, (0) Success
2015-04-10 09:05:21.887220 7fd476af6700 2 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).fault (0) Success
2015-04-10 09:05:21.887248 7fd476af6700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).fault already closed|closing
2015-04-10 09:05:21.887277 7fd476af6700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).reader done
2015-04-10 09:05:21.887328 7fd492dcf700 20 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).writer finishing
2015-04-10 09:05:21.887397 7fd492dcf700 10 -- 192.168.6.152:6805/3910 queue_reap 0x2373c80
2015-04-10 09:05:21.887416 7fd492dcf700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).writer done
2015-04-10 09:05:21.887442 7fd489a64700 10 -- 192.168.6.152:6805/3910 reaper
2015-04-10 09:05:21.887477 7fd489a64700 10 -- 192.168.6.152:6805/3910 reaper reaping pipe 0x2373c80 192.168.6.146:6789/0
2015-04-10 09:05:21.887487 7fd489a64700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).discard_queue
2015-04-10 09:05:21.887502 7fd489a64700 10 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).unregister_pipe - not registered
2015-04-10 09:05:21.887512 7fd489a64700 20 -- 192.168.6.152:6805/3910 >> 192.168.6.146:6789/0 pipe(0x2373c80 sd=25 :38403 s=4 pgs=2397361 cs=1 l=1 c=0x24779a0).join
2015-04-10 09:05:21.887752 7fd489a64700 10 -- 192.168.6.152:6805/3910 reaper reaped pipe 0x2373c80 192.168.6.146:6789/0
2015-04-10 09:05:21.887790 7fd489a64700 10 -- 192.168.6.152:6805/3910 reaper deleted pipe 0x2373c80
2015-04-10 09:05:21.887799 7fd489a64700 10 -- 192.168.6.152:6805/3910 reaper done
2015-04-10 09:05:21.888503 7fd4769f5700 10 -- 192.168.6.152:6805/3910 shutdown 192.168.6.152:6805/3910
2015-04-10 09:05:21.888527 7fd4769f5700 1 -- 192.168.6.152:6805/3910 mark_down_all
2015-04-10 09:05:21.888544 7fd4769f5700 10 -- 192.168.6.152:6806/3910 shutdown 192.168.6.152:6806/3910
2015-04-10 09:05:21.888555 7fd4769f5700 1 -- 192.168.6.152:6806/3910 mark_down_all
2015-04-10 09:05:21.888575 7fd4769f5700 10 -- :/3910 shutdown :/3910
2015-04-10 09:05:21.888584 7fd4769f5700 1 -- :/3910 mark_down_all
2015-04-10 09:05:21.888601 7fd4769f5700 10 -- 192.168.6.152:6809/3910 shutdown 192.168.6.152:6809/3910
2015-04-10 09:05:21.888611 7fd4769f5700 1 -- 192.168.6.152:6809/3910 mark_down_all
2015-04-10 09:05:21.888626 7fd4769f5700 10 -- 192.168.6.152:6808/3910 shutdown 192.168.6.152:6808/3910
2015-04-10 09:05:21.888636 7fd4769f5700 1 -- 192.168.6.152:6808/3910 mark_down_all
2015-04-10 09:05:21.888656 7fd4769f5700 10 -- 192.168.6.152:6807/3910 shutdown 192.168.6.152:6807/3910
2015-04-10 09:05:21.888666 7fd4769f5700 1 -- 192.168.6.152:6807/3910 mark_down_all
2015-04-10 09:05:21.888724 7fd492e0c780 10 -- 192.168.6.152:6805/3910 wait: dispatch queue is stopped
2015-04-10 09:05:21.888752 7fd492e0c780 20 -- 192.168.6.152:6805/3910 wait: stopping accepter thread
2015-04-10 09:05:21.888760 7fd492e0c780 10 accepter.stop accepter
2015-04-10 09:05:21.888823 7fd482b0e700 20 accepter.accepter poll got 1
2015-04-10 09:05:21.888840 7fd482b0e700 20 accepter.accepter closing
2015-04-10 09:05:21.888862 7fd482b0e700 10 accepter.accepter stopping
2015-04-10 09:05:21.888912 7fd492e0c780 20 -- 192.168.6.152:6805/3910 wait: stopped accepter thread
2015-04-10 09:05:21.888925 7fd492e0c780 20 -- 192.168.6.152:6805/3910 wait: stopping reaper thread
2015-04-10 09:05:21.888951 7fd489a64700 10 -- 192.168.6.152:6805/3910 reaper_entry done
2015-04-10 09:05:21.889024 7fd492e0c780 20 -- 192.168.6.152:6805/3910 wait: stopped reaper thread
2015-04-10 09:05:21.889038 7fd492e0c780 10 -- 192.168.6.152:6805/3910 wait: closing pipes
2015-04-10 09:05:21.889045 7fd492e0c780 10 -- 192.168.6.152:6805/3910 reaper
2015-04-10 09:05:21.889050 7fd492e0c780 10 -- 192.168.6.152:6805/3910 reaper done
2015-04-10 09:05:21.889054 7fd492e0c780 10 -- 192.168.6.152:6805/3910 wait: waiting for pipes to close
2015-04-10 09:05:21.889058 7fd492e0c780 10 -- 192.168.6.152:6805/3910 wait: done.
2015-04-10 09:05:21.889062 7fd492e0c780 1 -- 192.168.6.152:6805/3910 shutdown complete.
2015-04-10 09:05:21.889067 7fd492e0c780 10 -- :/3910 wait: waiting for dispatch queue
2015-04-10 09:05:21.889108 7fd492e0c780 10 -- :/3910 wait: dispatch queue is stopped
2015-04-10 09:05:21.889113 7fd492e0c780 20 -- :/3910 wait: stopping reaper thread
2015-04-10 09:05:21.889157 7fd48a265700 10 -- :/3910 reaper_entry done
2015-04-10 09:05:21.889240 7fd492e0c780 20 -- :/3910 wait: stopped reaper thread
2015-04-10 09:05:21.889248 7fd492e0c780 10 -- :/3910 wait: closing pipes
2015-04-10 09:05:21.889250 7fd492e0c780 10 -- :/3910 reaper
2015-04-10 09:05:21.889252 7fd492e0c780 10 -- :/3910 reaper done
2015-04-10 09:05:21.889255 7fd492e0c780 10 -- :/3910 wait: waiting for pipes to close
2015-04-10 09:05:21.889257 7fd492e0c780 10 -- :/3910 wait: done.
2015-04-10 09:05:21.889259 7fd492e0c780 1 -- :/3910 shutdown complete.
2015-04-10 09:05:21.889262 7fd492e0c780 10 -- 192.168.6.152:6808/3910 wait: waiting for dispatch queue
2015-04-10 09:05:21.889301 7fd492e0c780 10 -- 192.168.6.152:6808/3910 wait: dispatch queue is stopped
2015-04-10 09:05:21.889311 7fd492e0c780 20 -- 192.168.6.152:6808/3910 wait: stopping accepter thread
2015-04-10 09:05:21.889315 7fd492e0c780 10 accepter.stop accepter
2015-04-10 09:05:21.889361 7fd480309700 20 accepter.accepter poll got 1
2015-04-10 09:05:21.889370 7fd480309700 20 accepter.accepter closing
2015-04-10 09:05:21.889384 7fd480309700 10 accepter.accepter stopping
2015-04-10 09:05:21.889455 7fd492e0c780 20 -- 192.168.6.152:6808/3910 wait: stopped accepter thread
2015-04-10 09:05:21.889466 7fd492e0c780 20 -- 192.168.6.152:6808/3910 wait: stopping reaper thread
2015-04-10 09:05:21.889511 7fd48aa66700 10 -- 192.168.6.152:6808/3910 reaper_entry done
2015-04-10 09:05:21.889578 7fd492e0c780 20 -- 192.168.6.152:6808/3910 wait: stopped reaper thread
2015-04-10 09:05:21.889589 7fd492e0c780 10 -- 192.168.6.152:6808/3910 wait: closing pipes
2015-04-10 09:05:21.889592 7fd492e0c780 10 -- 192.168.6.152:6808/3910 reaper
2015-04-10 09:05:21.889596 7fd492e0c780 10 -- 192.168.6.152:6808/3910 reaper done
2015-04-10 09:05:21.889600 7fd492e0c780 10 -- 192.168.6.152:6808/3910 wait: waiting for pipes to close
2015-04-10 09:05:21.889604 7fd492e0c780 10 -- 192.168.6.152:6808/3910 wait: done.
2015-04-10 09:05:21.889609 7fd492e0c780 1 -- 192.168.6.152:6808/3910 shutdown complete.
2015-04-10 09:05:21.889613 7fd492e0c780 10 -- 192.168.6.152:6807/3910 wait: waiting for dispatch queue
2015-04-10 09:05:21.889638 7fd492e0c780 10 -- 192.168.6.152:6807/3910 wait: dispatch queue is stopped
2015-04-10 09:05:21.889646 7fd492e0c780 20 -- 192.168.6.152:6807/3910 wait: stopping accepter thread
2015-04-10 09:05:21.889650 7fd492e0c780 10 accepter.stop accepter
2015-04-10 09:05:21.889695 7fd47f307700 20 accepter.accepter poll got 1
2015-04-10 09:05:21.889704 7fd47f307700 20 accepter.accepter closing
2015-04-10 09:05:21.889715 7fd47f307700 10 accepter.accepter stopping
2015-04-10 09:05:21.889766 7fd492e0c780 20 -- 192.168.6.152:6807/3910 wait: stopped accepter thread
2015-04-10 09:05:21.889776 7fd492e0c780 20 -- 192.168.6.152:6807/3910 wait: stopping reaper thread
2015-04-10 09:05:21.889805 7fd48ca6a700 10 -- 192.168.6.152:6807/3910 reaper_entry done
2015-04-10 09:05:21.889869 7fd492e0c780 20 -- 192.168.6.152:6807/3910 wait: stopped reaper thread
2015-04-10 09:05:21.889879 7fd492e0c780 10 -- 192.168.6.152:6807/3910 wait: closing pipes
2015-04-10 09:05:21.889886 7fd492e0c780 10 -- 192.168.6.152:6807/3910 reaper
2015-04-10 09:05:21.889890 7fd492e0c780 10 -- 192.168.6.152:6807/3910 reaper done
2015-04-10 09:05:21.889894 7fd492e0c780 10 -- 192.168.6.152:6807/3910 wait: waiting for pipes to close
2015-04-10 09:05:21.889898 7fd492e0c780 10 -- 192.168.6.152:6807/3910 wait: done.
2015-04-10 09:05:21.889902 7fd492e0c780 1 -- 192.168.6.152:6807/3910 shutdown complete.
2015-04-10 09:05:21.889906 7fd492e0c780 10 -- 192.168.6.152:6806/3910 wait: waiting for dispatch queue
2015-04-10 09:05:21.889927 7fd492e0c780 10 -- 192.168.6.152:6806/3910 wait: dispatch queue is stopped
2015-04-10 09:05:21.889935 7fd492e0c780 20 -- 192.168.6.152:6806/3910 wait: stopping accepter thread
2015-04-10 09:05:21.889939 7fd492e0c780 10 accepter.stop accepter
2015-04-10 09:05:21.889985 7fd481b0c700 20 accepter.accepter poll got 1
2015-04-10 09:05:21.889994 7fd481b0c700 20 accepter.accepter closing
2015-04-10 09:05:21.890005 7fd481b0c700 10 accepter.accepter stopping
2015-04-10 09:05:21.890056 7fd492e0c780 20 -- 192.168.6.152:6806/3910 wait: stopped accepter thread
2015-04-10 09:05:21.890065 7fd492e0c780 20 -- 192.168.6.152:6806/3910 wait: stopping reaper thread
2015-04-10 09:05:21.890114 7fd48dd57700 10 -- 192.168.6.152:6806/3910 reaper_entry done
2015-04-10 09:05:21.890180 7fd492e0c780 20 -- 192.168.6.152:6806/3910 wait: stopped reaper thread
2015-04-10 09:05:21.890188 7fd492e0c780 10 -- 192.168.6.152:6806/3910 wait: closing pipes
2015-04-10 09:05:21.890192 7fd492e0c780 10 -- 192.168.6.152:6806/3910 reaper
2015-04-10 09:05:21.890196 7fd492e0c780 10 -- 192.168.6.152:6806/3910 reaper done
2015-04-10 09:05:21.890200 7fd492e0c780 10 -- 192.168.6.152:6806/3910 wait: waiting for pipes to close
2015-04-10 09:05:21.890204 7fd492e0c780 10 -- 192.168.6.152:6806/3910 wait: done.
2015-04-10 09:05:21.890208 7fd492e0c780 1 -- 192.168.6.152:6806/3910 shutdown complete.
2015-04-10 09:05:21.890212 7fd492e0c780 10 -- 192.168.6.152:6809/3910 wait: waiting for dispatch queue
2015-04-10 09:05:21.890233 7fd492e0c780 10 -- 192.168.6.152:6809/3910 wait: dispatch queue is stopped
2015-04-10 09:05:21.890243 7fd492e0c780 20 -- 192.168.6.152:6809/3910 wait: stopping accepter thread
2015-04-10 09:05:21.890247 7fd492e0c780 10 accepter.stop accepter
2015-04-10 09:05:21.890291 7fd47e305700 20 accepter.accepter poll got 1
2015-04-10 09:05:21.890300 7fd47e305700 20 accepter.accepter closing
2015-04-10 09:05:21.890312 7fd47e305700 10 accepter.accepter stopping
2015-04-10 09:05:21.890363 7fd492e0c780 20 -- 192.168.6.152:6809/3910 wait: stopped accepter thread
2015-04-10 09:05:21.890372 7fd492e0c780 20 -- 192.168.6.152:6809/3910 wait: stopping reaper thread
2015-04-10 09:05:21.890417 7fd48d556700 10 -- 192.168.6.152:6809/3910 reaper_entry done
2015-04-10 09:05:21.890487 7fd492e0c780 20 -- 192.168.6.152:6809/3910 wait: stopped reaper thread
2015-04-10 09:05:21.890496 7fd492e0c780 10 -- 192.168.6.152:6809/3910 wait: closing pipes
2015-04-10 09:05:21.890500 7fd492e0c780 10 -- 192.168.6.152:6809/3910 reaper
2015-04-10 09:05:21.890505 7fd492e0c780 10 -- 192.168.6.152:6809/3910 reaper done
2015-04-10 09:05:21.890511 7fd492e0c780 10 -- 192.168.6.152:6809/3910 wait: waiting for pipes to close
2015-04-10 09:05:21.890515 7fd492e0c780 10 -- 192.168.6.152:6809/3910 wait: done.
2015-04-10 09:05:21.890519 7fd492e0c780 1 -- 192.168.6.152:6809/3910 shutdown complete.
2015-04-10 09:05:24.574573 7f56997da780 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 1033
2015-04-10 09:05:24.579691 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) dump_stop
2015-04-10 09:05:24.580003 7f56997da780 5 filestore(/var/lib/ceph/osd/osd.15) basedir /var/lib/ceph/osd/osd.15 journal /dev/sdj4
2015-04-10 09:05:24.580064 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) mount fsid is e411ce75-c0f6-47f6-b58f-804a093fae81
2015-04-10 09:05:24.580187 7f56997da780 0 filestore(/var/lib/ceph/osd/osd.15) mount detected xfs (libxfs)
2015-04-10 09:05:24.580203 7f56997da780 1 filestore(/var/lib/ceph/osd/osd.15) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2015-04-10 09:05:24.583480 7f56997da780 0 genericfilestorebackend(/var/lib/ceph/osd/osd.15) detect_features: FIEMAP ioctl is supported and appears to work
2015-04-10 09:05:24.583510 7f56997da780 0 genericfilestorebackend(/var/lib/ceph/osd/osd.15) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-04-10 09:05:24.584973 7f56997da780 0 genericfilestorebackend(/var/lib/ceph/osd/osd.15) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-04-10 09:05:24.585185 7f56997da780 0 xfsfilestorebackend(/var/lib/ceph/osd/osd.15) detect_feature: extsize is disabled by conf
2015-04-10 09:05:24.586840 7f56997da780 5 filestore(/var/lib/ceph/osd/osd.15) mount op_seq is 30
2015-04-10 09:05:24.589549 7f56997da780 20 filestore (init)dbobjectmap: seq is 1
2015-04-10 09:05:24.589589 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) open_journal at /dev/sdj4
2015-04-10 09:05:24.589847 7f56997da780 0 filestore(/var/lib/ceph/osd/osd.15) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2015-04-10 09:05:24.589954 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) list_collections
2015-04-10 09:05:24.591374 7f569443c700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:24.594959 7f56997da780 1 journal _open /dev/sdj4 fd 21: 15999172608 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-04-10 09:05:24.599593 7f56997da780 1 journal _open /dev/sdj4 fd 21: 15999172608 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-04-10 09:05:24.600329 7f56997da780 5 filestore(/var/lib/ceph/osd/osd.15) umount /var/lib/ceph/osd/osd.15
2015-04-10 09:05:24.600372 7f569443c700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry force_sync set
2015-04-10 09:05:24.600758 7f56997da780 1 journal close /dev/sdj4
2015-04-10 09:05:24.602494 7f56997da780 10 ErasureCodePluginSelectJerasure: SSE4 plugin
2015-04-10 09:05:24.610876 7f56997da780 10 ErasureCodeJerasure: technique=reed_sol_van
2015-04-10 09:05:24.610904 7f56997da780 10 ErasureCodeJerasure: k defaults to 7
2015-04-10 09:05:24.610908 7f56997da780 10 ErasureCodeJerasure: m defaults to 3
2015-04-10 09:05:24.610911 7f56997da780 10 ErasureCodeJerasure: w defaults to 8
2015-04-10 09:05:24.610944 7f56997da780 10 load: jerasure
2015-04-10 09:05:24.611666 7f56997da780 5 filestore(/var/lib/ceph/osd/osd.15) test_mount basedir /var/lib/ceph/osd/osd.15 journal /dev/sdj4
2015-04-10 09:05:24.611929 7f56997da780 2 osd.15 0 mounting /var/lib/ceph/osd/osd.15 /dev/sdj4
2015-04-10 09:05:24.611937 7f56997da780 5 filestore(/var/lib/ceph/osd/osd.15) basedir /var/lib/ceph/osd/osd.15 journal /dev/sdj4
2015-04-10 09:05:24.611955 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) mount fsid is e411ce75-c0f6-47f6-b58f-804a093fae81
2015-04-10 09:05:24.611992 7f56997da780 0 filestore(/var/lib/ceph/osd/osd.15) mount detected xfs (libxfs)
2015-04-10 09:05:24.613749 7f56997da780 0 genericfilestorebackend(/var/lib/ceph/osd/osd.15) detect_features: FIEMAP ioctl is supported and appears to work
2015-04-10 09:05:24.613760 7f56997da780 0 genericfilestorebackend(/var/lib/ceph/osd/osd.15) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-04-10 09:05:24.614540 7f56997da780 0 genericfilestorebackend(/var/lib/ceph/osd/osd.15) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-04-10 09:05:24.614633 7f56997da780 0 xfsfilestorebackend(/var/lib/ceph/osd/osd.15) detect_feature: extsize is disabled by conf
2015-04-10 09:05:24.615472 7f56997da780 5 filestore(/var/lib/ceph/osd/osd.15) mount op_seq is 30
2015-04-10 09:05:24.616855 7f56997da780 20 filestore (init)dbobjectmap: seq is 1
2015-04-10 09:05:24.616869 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) open_journal at /dev/sdj4
2015-04-10 09:05:24.616891 7f56997da780 0 filestore(/var/lib/ceph/osd/osd.15) mount: WRITEAHEAD journal mode explicitly enabled in conf
2015-04-10 09:05:24.616895 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) list_collections
2015-04-10 09:05:24.617041 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:24.619712 7f56997da780 1 journal _open /dev/sdj4 fd 22: 15999172608 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-04-10 09:05:24.622364 7f56997da780 1 journal _open /dev/sdj4 fd 22: 15999172608 bytes, block size 4096 bytes, directio = 1, aio = 1
2015-04-10 09:05:24.622883 7f56997da780 2 osd.15 0 boot
2015-04-10 09:05:24.622907 7f56997da780 15 filestore(/var/lib/ceph/osd/osd.15) read meta/23c2fcde/osd_superblock/0//-1 0~0
2015-04-10 09:05:24.623091 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) FileStore::read meta/23c2fcde/osd_superblock/0//-1 0~358/358
2015-04-10 09:05:24.623124 7f56997da780 10 osd.15 0 read_superblock sb(71783e7e-b856-4b96-a478-27191a56bf17 osd.15 e411ce75-c0f6-47f6-b58f-804a093fae81 e3814 [2527,3814] lci=[0,3814])
2015-04-10 09:05:24.623252 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) stat meta/16ef7597/infos/head//-1 = 0 (size 0)
2015-04-10 09:05:24.623309 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) stat meta/a468ec03/snapmapper/0//-1 = 0 (size 0)
2015-04-10 09:05:24.623331 7f56997da780 10 open_all_classes
2015-04-10 09:05:24.623434 7f56997da780 10 open_all_classes found statelog
2015-04-10 09:05:24.623456 7f56997da780 10 _get_class adding new class name statelog 0x19311e8
2015-04-10 09:05:24.623466 7f56997da780 10 _load_class statelog from /usr/lib/rados-classes/libcls_statelog.so
2015-04-10 09:05:24.623840 7f56997da780 10 register_class statelog status 3
2015-04-10 09:05:24.623843 7f56997da780 10 register_cxx_method statelog.add flags 3 0x7f569351c290
2015-04-10 09:05:24.623848 7f56997da780 10 register_cxx_method statelog.list flags 1 0x7f569351c5b0
2015-04-10 09:05:24.623852 7f56997da780 10 register_cxx_method statelog.remove flags 3 0x7f569351b210
2015-04-10 09:05:24.623857 7f56997da780 10 register_cxx_method statelog.check_state flags 1 0x7f569351ba00
2015-04-10 09:05:24.623861 7f56997da780 10 _load_class statelog success
2015-04-10 09:05:24.623864 7f56997da780 10 open_all_classes found hello
2015-04-10 09:05:24.623868 7f56997da780 10 _get_class adding new class name hello 0x1931108
2015-04-10 09:05:24.623871 7f56997da780 10 _load_class hello from /usr/lib/rados-classes/libcls_hello.so
2015-04-10 09:05:24.624101 7f56997da780 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
2015-04-10 09:05:24.624114 7f56997da780 10 register_class hello status 3
2015-04-10 09:05:24.624116 7f56997da780 10 register_cxx_method hello.say_hello flags 1 0x7f568b221f50
2015-04-10 09:05:24.624120 7f56997da780 10 register_cxx_method hello.record_hello flags 2 0x7f568b222f90
2015-04-10 09:05:24.624124 7f56997da780 10 register_cxx_method hello.writes_dont_return_data flags 2 0x7f568b222270
2015-04-10 09:05:24.624129 7f56997da780 10 register_cxx_method hello.replay flags 1 0x7f568b221f20
2015-04-10 09:05:24.624132 7f56997da780 10 register_cxx_method hello.turn_it_to_11 flags 3 0x7f568b222510
2015-04-10 09:05:24.624136 7f56997da780 10 register_cxx_method hello.bad_reader flags 2 0x7f568b221f10
2015-04-10 09:05:24.624139 7f56997da780 10 register_cxx_method hello.bad_writer flags 1 0x7f568b221f00
2015-04-10 09:05:24.624142 7f56997da780 10 _load_class hello success
2015-04-10 09:05:24.624145 7f56997da780 10 open_all_classes found refcount
2015-04-10 09:05:24.624149 7f56997da780 10 _get_class adding new class name refcount 0x1931488
2015-04-10 09:05:24.624152 7f56997da780 10 _load_class refcount from /usr/lib/rados-classes/libcls_refcount.so
2015-04-10 09:05:24.627612 7f56997da780 10 register_class refcount status 3
2015-04-10 09:05:24.627614 7f56997da780 10 register_cxx_method refcount.get flags 3 0x7f568af657d0
2015-04-10 09:05:24.627618 7f56997da780 10 register_cxx_method refcount.put flags 3 0x7f568af646d0
2015-04-10 09:05:24.627624 7f56997da780 10 register_cxx_method refcount.set flags 3 0x7f568af65260
2015-04-10 09:05:24.627627 7f56997da780 10 register_cxx_method refcount.read flags 1 0x7f568af64d20
2015-04-10 09:05:24.627632 7f56997da780 10 _load_class refcount success
2015-04-10 09:05:24.627649 7f56997da780 10 open_all_classes found version
2015-04-10 09:05:24.627653 7f56997da780 10 _get_class adding new class name version 0x19313a8
2015-04-10 09:05:24.627656 7f56997da780 10 _load_class version from /usr/lib/rados-classes/libcls_version.so
2015-04-10 09:05:24.627902 7f56997da780 10 register_class version status 3
2015-04-10 09:05:24.627904 7f56997da780 10 register_cxx_method version.set flags 3 0x7f568acc22a0
2015-04-10 09:05:24.627909 7f56997da780 10 register_cxx_method version.inc flags 3 0x7f568acc2f30
2015-04-10 09:05:24.627912 7f56997da780 10 register_cxx_method version.inc_conds flags 3 0x7f568acc2f30
2015-04-10 09:05:24.627916 7f56997da780 10 register_cxx_method version.read flags 1 0x7f568acc2ae0
2015-04-10 09:05:24.627919 7f56997da780 10 register_cxx_method version.check_conds flags 1 0x7f568acc38d0
2015-04-10 09:05:24.627923 7f56997da780 10 _load_class version success
2015-04-10 09:05:24.627925 7f56997da780 10 open_all_classes found lock
2015-04-10 09:05:24.627929 7f56997da780 10 _get_class adding new class name lock 0x19312c8
2015-04-10 09:05:24.627933 7f56997da780 10 _load_class lock from /usr/lib/rados-classes/libcls_lock.so
2015-04-10 09:05:24.628139 7f56997da780 10 register_class lock status 3
2015-04-10 09:05:24.628143 7f56997da780 10 register_cxx_method lock.lock flags 3 0x7f568aab3d80
2015-04-10 09:05:24.628147 7f56997da780 10 register_cxx_method lock.unlock flags 3 0x7f568aab53f0
2015-04-10 09:05:24.628150 7f56997da780 10 register_cxx_method lock.break_lock flags 3 0x7f568aab5800
2015-04-10 09:05:24.628153 7f56997da780 10 register_cxx_method lock.get_info flags 1 0x7f568aab5bb0
2015-04-10 09:05:24.628167 7f56997da780 10 register_cxx_method lock.list_locks flags 1 0x7f568aab32f0
2015-04-10 09:05:24.628170 7f56997da780 10 _load_class lock success
2015-04-10 09:05:24.628173 7f56997da780 10 open_all_classes found rbd
2015-04-10 09:05:24.628177 7f56997da780 10 _get_class adding new class name rbd 0x1931808
2015-04-10 09:05:24.628180 7f56997da780 10 _load_class rbd from /usr/lib/rados-classes/libcls_rbd.so
2015-04-10 09:05:24.628507 7f56997da780 10 register_class rbd status 3
2015-04-10 09:05:24.628511 7f56997da780 10 register_cxx_method rbd.create flags 3 0x7f568a89ae30
2015-04-10 09:05:24.628514 7f56997da780 10 register_cxx_method rbd.get_features flags 1 0x7f568a89c290
2015-04-10 09:05:24.628518 7f56997da780 10 register_cxx_method rbd.get_size flags 1 0x7f568a89c660
2015-04-10 09:05:24.628530 7f56997da780 10 register_cxx_method rbd.set_size flags 3 0x7f568a899b60
2015-04-10 09:05:24.628534 7f56997da780 10 register_cxx_method rbd.get_snapcontext flags 1 0x7f568a89b8c0
2015-04-10 09:05:24.628539 7f56997da780 10 register_cxx_method rbd.get_object_prefix flags 1 0x7f568a8978a0
2015-04-10 09:05:24.628542 7f56997da780 10 register_cxx_method rbd.get_snapshot_name flags 1 0x7f568a89d7a0
2015-04-10 09:05:24.628546 7f56997da780 10 register_cxx_method rbd.snapshot_add flags 3 0x7f568a89ef00
2015-04-10 09:05:24.628549 7f56997da780 10 register_cxx_method rbd.snapshot_remove flags 3 0x7f568a89da80
2015-04-10 09:05:24.628552 7f56997da780 10 register_cxx_method rbd.get_all_features flags 1 0x7f568a895750
2015-04-10 09:05:24.628556 7f56997da780 10 register_cxx_method rbd.copyup flags 3 0x7f568a895680
2015-04-10 09:05:24.628559 7f56997da780 10 register_cxx_method rbd.get_parent flags 1 0x7f568a89d360
2015-04-10 09:05:24.628562 7f56997da780 10 register_cxx_method rbd.set_parent flags 3 0x7f568a89a300
2015-04-10 09:05:24.628575 7f56997da780 10 register_cxx_method rbd.remove_parent flags 3 0x7f568a899180
2015-04-10 09:05:24.628578 7f56997da780 10 register_cxx_method rbd.set_protection_status flags 3 0x7f568a89cea0
2015-04-10 09:05:24.628593 7f56997da780 10 register_cxx_method rbd.get_protection_status flags 1 0x7f568a89cb40
2015-04-10 09:05:24.628598 7f56997da780 10 register_cxx_method rbd.get_stripe_unit_count flags 1 0x7f568a898b10
2015-04-10 09:05:24.628601 7f56997da780 10 register_cxx_method rbd.set_stripe_unit_count flags 3 0x7f568a899610
2015-04-10 09:05:24.628605 7f56997da780 10 register_cxx_method rbd.add_child flags 3 0x7f568a89e010
2015-04-10 09:05:24.628609 7f56997da780 10 register_cxx_method rbd.remove_child flags 3 0x7f568a89e8c0
2015-04-10 09:05:24.628616 7f56997da780 10 register_cxx_method rbd.get_children flags 1 0x7f568a89e580
2015-04-10 09:05:24.628620 7f56997da780 10 register_cxx_method rbd.get_id flags 1 0x7f568a899360
2015-04-10 09:05:24.628623 7f56997da780 10 register_cxx_method rbd.set_id flags 3 0x7f568a89ab30
2015-04-10 09:05:24.628626 7f56997da780 10 register_cxx_method rbd.dir_get_id flags 1 0x7f568a897a70
2015-04-10 09:05:24.628629 7f56997da780 10 register_cxx_method rbd.dir_get_name flags 1 0x7f568a897d60
2015-04-10 09:05:24.628634 7f56997da780 10 register_cxx_method rbd.dir_list flags 1 0x7f568a8a0f00
2015-04-10 09:05:24.628637 7f56997da780 10 register_cxx_method rbd.dir_add_image flags 3 0x7f568a8a0cd0
2015-04-10 09:05:24.628641 7f56997da780 10 register_cxx_method rbd.dir_remove_image flags 3 0x7f568a898470
2015-04-10 09:05:24.628644 7f56997da780 10 register_cxx_method rbd.dir_rename_image flags 3 0x7f568a8a09e0
2015-04-10 09:05:24.628648 7f56997da780 10 register_cxx_method rbd.snap_list flags 1 0x7f568a8965e0
2015-04-10 09:05:24.628651 7f56997da780 10 register_cxx_method rbd.snap_add flags 3 0x7f568a897040
2015-04-10 09:05:24.628655 7f56997da780 10 register_cxx_method rbd.snap_remove flags 3 0x7f568a8968c0
2015-04-10 09:05:24.628668 7f56997da780 10 _load_class rbd success
2015-04-10 09:05:24.628672 7f56997da780 10 open_all_classes found replica_log
2015-04-10 09:05:24.628675 7f56997da780 10 _get_class adding new class name replica_log 0x1931728
2015-04-10 09:05:24.628678 7f56997da780 10 _load_class replica_log from /usr/lib/rados-classes/libcls_replica_log.so
2015-04-10 09:05:24.628875 7f56997da780 10 register_class replica_log status 3
2015-04-10 09:05:24.628877 7f56997da780 10 register_cxx_method replica_log.set flags 3 0x7f568a68cf50
2015-04-10 09:05:24.628881 7f56997da780 10 register_cxx_method replica_log.get flags 1 0x7f568a68bab0
2015-04-10 09:05:24.628885 7f56997da780 10 register_cxx_method replica_log.delete flags 3 0x7f568a68b4d0
2015-04-10 09:05:24.628889 7f56997da780 10 _load_class replica_log success
2015-04-10 09:05:24.628892 7f56997da780 10 open_all_classes found user
2015-04-10 09:05:24.628895 7f56997da780 10 _get_class adding new class name user 0x1931568
2015-04-10 09:05:24.628898 7f56997da780 10 _load_class user from /usr/lib/rados-classes/libcls_user.so
2015-04-10 09:05:24.629088 7f56997da780 10 register_class user status 3
2015-04-10 09:05:24.629090 7f56997da780 10 register_cxx_method user.set_buckets_info flags 3 0x7f568a47cd60
2015-04-10 09:05:24.629094 7f56997da780 10 register_cxx_method user.complete_stats_sync flags 3 0x7f568a47a800
2015-04-10 09:05:24.629097 7f56997da780 10 register_cxx_method user.remove_bucket flags 3 0x7f568a47bf50
2015-04-10 09:05:24.629101 7f56997da780 10 register_cxx_method user.list_buckets flags 1 0x7f568a47e240
2015-04-10 09:05:24.629113 7f56997da780 10 register_cxx_method user.get_header flags 1 0x7f568a47aee0
2015-04-10 09:05:24.629117 7f56997da780 10 _load_class user success
2015-04-10 09:05:24.629120 7f56997da780 10 open_all_classes found rgw
2015-04-10 09:05:24.629124 7f56997da780 10 _get_class adding new class name rgw 0x1931c68
2015-04-10 09:05:24.629127 7f56997da780 10 _load_class rgw from /usr/lib/rados-classes/libcls_rgw.so
2015-04-10 09:05:24.631795 7f56997da780 10 register_class rgw status 3
2015-04-10 09:05:24.631798 7f56997da780 10 register_cxx_method rgw.bucket_init_index flags 3 0x7f568a1aa150
2015-04-10 09:05:24.631802 7f56997da780 10 register_cxx_method rgw.bucket_set_tag_timeout flags 3 0x7f568a1acb70
2015-04-10 09:05:24.631806 7f56997da780 10 register_cxx_method rgw.bucket_list flags 1 0x7f568a1ae640
2015-04-10 09:05:24.631810 7f56997da780 10 register_cxx_method rgw.bucket_check_index flags 1 0x7f568a1afdf0
2015-04-10 09:05:24.631814 7f56997da780 10 register_cxx_method rgw.bucket_rebuild_index flags 3 0x7f568a1b0040
2015-04-10 09:05:24.631817 7f56997da780 10 register_cxx_method rgw.bucket_prepare_op flags 3 0x7f568a1ade10
2015-04-10 09:05:24.631821 7f56997da780 10 register_cxx_method rgw.bucket_complete_op flags 3 0x7f568a1b0340
2015-04-10 09:05:24.631824 7f56997da780 10 register_cxx_method rgw.bi_log_list flags 1 0x7f568a1ac2f0
2015-04-10 09:05:24.631828 7f56997da780 10 register_cxx_method rgw.bi_log_trim flags 3 0x7f568a1ac690
2015-04-10 09:05:24.631832 7f56997da780 10 register_cxx_method rgw.dir_suggest_changes flags 3 0x7f568a1ad030
2015-04-10 09:05:24.631835 7f56997da780 10 register_cxx_method rgw.user_usage_log_add flags 3 0x7f568a1b1560
2015-04-10 09:05:24.631839 7f56997da780 10 register_cxx_method rgw.user_usage_log_read flags 1 0x7f568a1ab320
2015-04-10 09:05:24.631853 7f56997da780 10 register_cxx_method rgw.user_usage_log_trim flags 3 0x7f568a1ab580
2015-04-10 09:05:24.631857 7f56997da780 10 register_cxx_method rgw.gc_set_entry flags 3 0x7f568a1a9860
2015-04-10 09:05:24.631869 7f56997da780 10 register_cxx_method rgw.gc_defer_entry flags 3 0x7f568a1a9aa0
2015-04-10 09:05:24.631873 7f56997da780 10 register_cxx_method rgw.gc_list flags 1 0x7f568a1aa430
2015-04-10 09:05:24.631877 7f56997da780 10 register_cxx_method rgw.gc_remove flags 3 0x7f568a1a9da0
2015-04-10 09:05:24.631880 7f56997da780 10 _load_class rgw success
2015-04-10 09:05:24.631884 7f56997da780 10 open_all_classes found log
2015-04-10 09:05:24.631888 7f56997da780 10 _get_class adding new class name log 0x1931b88
2015-04-10 09:05:24.631892 7f56997da780 10 _load_class log from /usr/lib/rados-classes/libcls_log.so
2015-04-10 09:05:24.632121 7f56997da780 10 register_class log status 3
2015-04-10 09:05:24.632123 7f56997da780 10 register_cxx_method log.add flags 3 0x7f5689efaec0
2015-04-10 09:05:24.632127 7f56997da780 10 register_cxx_method log.list flags 1 0x7f5689efb8a0
2015-04-10 09:05:24.632130 7f56997da780 10 register_cxx_method log.trim flags 3 0x7f5689efcf20
2015-04-10 09:05:24.632134 7f56997da780 10 register_cxx_method log.info flags 1 0x7f5689efa9c0
2015-04-10 09:05:24.632137 7f56997da780 10 _load_class log success
2015-04-10 09:05:24.632140 7f56997da780 10 open_all_classes found kvs
2015-04-10 09:05:24.632144 7f56997da780 10 _get_class adding new class name kvs 0x1931aa8
2015-04-10 09:05:24.632147 7f56997da780 10 _load_class kvs from /usr/lib/rados-classes/libcls_kvs.so
2015-04-10 09:05:24.632415 7f56997da780 10 register_class kvs status 3
2015-04-10 09:05:24.632422 7f56997da780 10 register_cxx_method kvs.get_idata_from_key flags 1 0x7f5689ce4b70
2015-04-10 09:05:24.632435 7f56997da780 10 register_cxx_method kvs.get_next_idata flags 1 0x7f5689ce6800
2015-04-10 09:05:24.632439 7f56997da780 10 register_cxx_method kvs.get_prev_idata flags 1 0x7f5689ce77f0
2015-04-10 09:05:24.632443 7f56997da780 10 register_cxx_method kvs.read_many flags 1 0x7f5689ce9cc0
2015-04-10 09:05:24.632446 7f56997da780 10 register_cxx_method kvs.check_writable flags 3 0x7f5689ce4730
2015-04-10 09:05:24.632449 7f56997da780 10 register_cxx_method kvs.assert_size_in_bound flags 2 0x7f5689ce4970
2015-04-10 09:05:24.632452 7f56997da780 10 register_cxx_method kvs.omap_insert flags 2 0x7f5689cec0d0
2015-04-10 09:05:24.632456 7f56997da780 10 register_cxx_method kvs.create_with_omap flags 2 0x7f5689ceb810
2015-04-10 09:05:24.632459 7f56997da780 10 register_cxx_method kvs.omap_remove flags 2 0x7f5689ce90a0
2015-04-10 09:05:24.632462 7f56997da780 10 register_cxx_method kvs.maybe_read_for_balance flags 1 0x7f5689cea8c0
2015-04-10 09:05:24.632469 7f56997da780 10 _load_class kvs success
2015-04-10 09:05:24.632511 7f56997da780 20 osd.15 0 get_map 3814 - loading and decoding 0x193c200
2015-04-10 09:05:24.632526 7f56997da780 15 filestore(/var/lib/ceph/osd/osd.15) read meta/a080e38/osdmap.3814/0//-1 0~0
2015-04-10 09:05:24.632738 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) FileStore::read meta/a080e38/osdmap.3814/0//-1 0~38606/38606
2015-04-10 09:05:24.632749 7f56997da780 10 osd.15 0 add_map_bl 3814 38606 bytes
2015-04-10 09:05:24.633910 7f56997da780 0 osd.15 3814 crush map has features 1107558400, adjusting msgr requires for clients
2015-04-10 09:05:24.633940 7f56997da780 0 osd.15 3814 crush map has features 1107558400 was 8705, adjusting msgr requires for mons
2015-04-10 09:05:24.633957 7f56997da780 0 osd.15 3814 crush map has features 1107558400, adjusting msgr requires for osds
2015-04-10 09:05:24.633979 7f56997da780 10 osd.15 3814 create_recoverystate_perf
2015-04-10 09:05:24.634004 7f56997da780 0 osd.15 3814 load_pgs
2015-04-10 09:05:24.634012 7f56997da780 10 filestore(/var/lib/ceph/osd/osd.15) list_collections
2015-04-10 09:05:24.634099 7f56997da780 10 osd.15 3814 load_pgs ignoring unrecognized meta
2015-04-10 09:05:24.634105 7f56997da780 0 osd.15 3814 load_pgs opened 0 pgs
2015-04-10 09:05:24.634115 7f56997da780 10 osd.15 3814 build_past_intervals_parallel nothing to build
2015-04-10 09:05:24.634124 7f56997da780 2 osd.15 3814 superblock: i am osd.15
2015-04-10 09:05:24.634128 7f56997da780 10 osd.15 3814 create_logger
2015-04-10 09:05:24.635576 7f56997da780 10 osd.15 3814 set_disk_tp_priority class priority -1
2015-04-10 09:05:24.635589 7f56997da780 -1 osd.15 3814 set_disk_tp_priority(22) Invalid argument: osd_disk_thread_ioprio_class is but only the following values are allowed: idle, be or rt
2015-04-10 09:05:24.635708 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:24.635741 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:24.635912 7f567dcc7700 10 osd.15 3814 agent_entry start
2015-04-10 09:05:24.635936 7f567dcc7700 20 osd.15 3814 agent_entry empty queue
2015-04-10 09:05:24.637082 7f569979f700 10 osd.15 3814 OSD::ms_get_authorizer type=mon
2015-04-10 09:05:24.637506 7f5689cdf700 10 osd.15 3814 ms_handle_connect on mon
2015-04-10 09:05:24.637541 7f5689cdf700 10 osd.15 3814 send_alive up_thru currently 2773 want 0
2015-04-10 09:05:24.637553 7f5689cdf700 20 osd.15 3814 send_pg_stats
2015-04-10 09:05:24.644057 7f5689cdf700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:24.644070 7f5689cdf700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:24.644075 7f5689cdf700 20 osd.15 3814 _dispatch 0x1904240 osd_map(3814..3814 src has 2527..3814) v3
2015-04-10 09:05:24.644102 7f5689cdf700 0 osd.15 3814 ignoring osdmap until we have initialized
2015-04-10 09:05:24.644107 7f5689cdf700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:24.644110 7f5689cdf700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:24.644608 7f5689cdf700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:24.644621 7f5689cdf700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:24.644625 7f5689cdf700 20 osd.15 3814 _dispatch 0x1a45d80 osd_map(3814..3814 src has 2527..3814) v3
2015-04-10 09:05:24.644631 7f5689cdf700 0 osd.15 3814 ignoring osdmap until we have initialized
2015-04-10 09:05:24.644635 7f5689cdf700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:24.644637 7f5689cdf700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:24.645081 7f56997da780 10 osd.15 3814 ensuring pgs have consumed prior maps
2015-04-10 09:05:24.645084 7f56997da780 7 osd.15 3814 consume_map version 3814
2015-04-10 09:05:24.645091 7f56997da780 0 osd.15 3814 done with init, starting boot process
2015-04-10 09:05:24.645094 7f56997da780 10 osd.15 3814 start_boot - have maps 2527..3814
2015-04-10 09:05:24.646203 7f5683cd3700 10 osd.15 3814 _maybe_boot mon has osdmaps 2527..3814
2015-04-10 09:05:24.646242 7f5683cd3700 10 osd.15 3814 _send_boot
2015-04-10 09:05:24.646247 7f5683cd3700 10 osd.15 3814 assuming cluster_addr ip matches client_addr
2015-04-10 09:05:24.646249 7f5683cd3700 10 osd.15 3814 assuming hb_back_addr ip matches cluster_addr
2015-04-10 09:05:24.646264 7f5683cd3700 10 osd.15 3814 client_addr 192.168.6.152:6805/1033, cluster_addr 192.168.6.152:6806/1033, hb_back_addr 192.168.6.152:6807/1033, hb_front_addr 192.168.6.152:6808/1033
2015-04-10 09:05:24.685325 7f5683cd3700 10 osd.15 3814 _collect_metadata {arch=x86_64,back_addr=192.168.6.152:6806/1033,ceph_version=ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047),cpu=AMD Opteron(tm) Processor 6320,distro=Ubuntu,distro_codename=precise,distro_description=Ubuntu 12.04.5 LTS,distro_version=12.04,front_addr=192.168.6.152:6805/1033,hb_back_addr=192.168.6.152:6807/1033,hb_front_addr=192.168.6.152:6808/1033,hostname=ceph02,kernel_description=#62~precise1-Ubuntu SMP Mon Aug 18 14:52:04 UTC 2014,kernel_version=3.13.0-35-generic,mem_swap_kb=7761400,mem_total_kb=36960444,os=Linux,osd_data=/var/lib/ceph/osd/osd.15,osd_journal=/dev/sdj4}
2015-04-10 09:05:29.490780 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:29.490820 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:29.740693 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.123648
2015-04-10 09:05:29.740741 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:30.740659 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:30.740699 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:30.740704 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:31.740795 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:31.740834 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:31.740840 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:32.740901 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:32.740951 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:32.740958 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:32.990976 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:32.991006 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:33.741040 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:33.741070 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:33.741078 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:34.740830 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000086
2015-04-10 09:05:34.740874 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:34.741154 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:34.741180 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:34.741184 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:35.741249 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:35.741268 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:35.741272 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:36.741344 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:36.741365 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:36.741368 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:37.177850 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:37.177882 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:37.741438 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:37.741462 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:37.741465 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:38.741551 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:38.741575 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:38.741580 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:38.945238 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:38.945268 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:39.740958 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000081
2015-04-10 09:05:39.741010 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:39.741659 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:39.741678 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:39.741682 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:40.741802 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:40.741830 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:40.741837 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:41.741942 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:41.741964 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:41.741968 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:42.742180 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:42.742203 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:42.742206 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:43.742416 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:43.742452 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:43.742456 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:44.741083 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000072
2015-04-10 09:05:44.741113 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:44.742659 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:44.742694 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:44.742698 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:44.991741 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:44.991770 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:45.742807 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:45.742846 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:45.742853 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:46.240858 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:46.240893 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:46.743063 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:46.743103 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:46.743109 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:47.744698 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:47.744723 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:47.744729 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:47.992142 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:47.992171 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:48.990817 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:48.990843 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:48.990848 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:49.741188 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000072
2015-04-10 09:05:49.741228 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:49.990924 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:49.990948 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:49.990953 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:50.991027 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:50.991051 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:50.991057 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:51.991160 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:51.991206 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:51.991212 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:52.692485 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:52.692533 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:52.991285 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:52.991314 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:52.991318 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:53.192840 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:53.192904 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:53.991542 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:53.991581 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:53.991589 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:54.741307 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000078
2015-04-10 09:05:54.741344 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:54.991672 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:54.991700 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:54.991705 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:55.991790 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:55.991815 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:55.991820 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:56.991897 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:56.991933 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:56.991938 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:57.992026 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:57.992061 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:57.992066 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:58.992225 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:58.992316 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:58.992319 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:05:59.093205 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:59.093256 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:05:59.741424 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000079
2015-04-10 09:05:59.741484 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:05:59.992426 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:05:59.992457 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:05:59.992462 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:01.005009 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:01.005061 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:01.005065 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:01.393639 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:01.393689 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:02.005278 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:02.005312 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:02.005316 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:03.005540 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:03.005584 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:03.005592 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:04.005810 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:04.005855 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:04.005861 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:04.741564 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000080
2015-04-10 09:06:04.741605 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:05.031718 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:05.031756 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:05.031761 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:05.493991 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:05.494031 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:05.994341 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:05.994392 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:06.031853 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:06.031886 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:06.031889 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:07.094251 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:07.094284 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:07.094290 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:08.240926 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:08.240963 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:08.240973 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:09.241053 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:09.241084 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:09.241090 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:09.741715 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000109
2015-04-10 09:06:09.741765 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:10.241012 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:10.241059 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:10.241176 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:10.241193 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:10.241197 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:11.241281 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:11.241312 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:11.241318 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:11.941220 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:11.941242 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:12.241399 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:12.241426 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:12.241430 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:13.241600 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:13.241622 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:13.241626 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:14.241740 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:14.241766 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:14.241771 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:14.741838 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000073
2015-04-10 09:06:14.741875 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:15.241934 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:15.241975 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:15.241979 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:16.241041 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:16.241072 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:16.242085 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:16.242112 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:16.242117 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:17.242214 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:17.242244 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:17.242248 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:17.491087 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:17.491121 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:18.242463 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:18.242510 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:18.242514 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:19.352455 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:19.352478 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:19.352481 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:19.742001 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000125
2015-04-10 09:06:19.742061 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:20.352551 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:20.352572 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:20.352577 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:21.352654 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:21.352671 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:21.352676 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:22.352831 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:22.352856 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:22.352861 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:22.990994 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:22.991026 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:23.490948 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:23.490978 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:23.490984 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:24.491066 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:24.491097 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:24.491103 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:24.742127 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000066
2015-04-10 09:06:24.742153 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:25.491203 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:25.491239 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:25.491244 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:26.491293 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:26.491319 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:26.491324 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:27.491502 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:27.491522 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:27.491526 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:27.741078 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:27.741121 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:28.491723 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:28.491755 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:28.491760 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:29.491834 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:29.491858 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:29.491866 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:29.742252 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000098
2015-04-10 09:06:29.742293 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:30.491946 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:30.491972 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:30.491977 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:31.492061 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:31.492095 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:31.492111 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:32.492329 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:32.492363 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:32.492367 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:33.492459 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:33.492480 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:33.492484 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:33.741108 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:33.741141 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:34.492563 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:34.492587 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:34.492592 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:34.742369 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000076
2015-04-10 09:06:34.742402 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:35.491100 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:35.491127 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:35.492729 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:35.492750 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:35.492755 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:35.991265 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:35.991293 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:36.528438 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:36.528464 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:36.528470 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:37.528680 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:37.528706 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:37.528710 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:37.741096 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:37.741124 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:38.741331 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:38.741359 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:38.741364 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:39.741439 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:39.741458 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:39.741463 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:39.742479 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000075
2015-04-10 09:06:39.742540 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:40.241162 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:40.241193 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:40.741573 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:40.741602 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:40.741611 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:41.741888 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:41.741912 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:41.741917 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:42.741990 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:42.742011 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:42.742020 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:43.742261 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:43.742302 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:43.742306 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:44.742387 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:44.742413 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:44.742421 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:44.742616 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000075
2015-04-10 09:06:44.742659 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:44.991155 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:44.991182 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:45.742506 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:45.742543 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:45.742550 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:46.742624 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:46.742647 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:46.742652 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:47.742730 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:47.742767 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:47.742771 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:48.491321 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:48.491373 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:48.742877 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:48.742901 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:48.742904 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:49.762412 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:49.762437 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:49.762442 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:49.762448 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.019788
2015-04-10 09:06:49.762480 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:50.991083 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:50.991120 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:50.991125 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:51.991305 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:51.991346 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:51.991380 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:52.991462 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:52.991501 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:52.991506 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:53.991591 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:53.991608 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:53.991612 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:54.491186 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:54.491226 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:54.991141 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.228659
2015-04-10 09:06:54.991180 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:54.991642 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:54.991654 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:54.991656 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:55.991756 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:55.991786 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:55.991791 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:56.241201 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:56.241232 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:56.991905 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:56.991937 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:56.991941 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:57.941565 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:57.941598 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:06:57.992210 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:57.992270 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:57.992285 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:58.992365 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:58.992393 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:58.992397 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:06:59.991258 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000077
2015-04-10 09:06:59.991307 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:06:59.992595 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:06:59.992633 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:06:59.992637 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:00.992718 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:00.992741 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:00.992745 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:01.992832 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:01.992857 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:01.992861 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:02.993077 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:02.993109 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:02.993114 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:03.241809 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:03.241842 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:03.993337 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:03.993372 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:03.993378 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:04.991388 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000079
2015-04-10 09:07:04.991438 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:04.993608 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:04.993649 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:04.993654 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:05.993992 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:05.994018 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:05.994024 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:06.996750 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:06.996775 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:06.996782 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:08.241192 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:08.241219 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:08.241225 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:09.157423 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:09.157454 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:09.241309 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:09.241338 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:09.241345 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:09.991519 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000080
2015-04-10 09:07:09.991574 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:10.241453 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:10.241483 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:10.241489 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:11.241596 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:11.241649 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:11.241653 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:12.241765 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:12.241809 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:12.241814 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:13.241974 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:13.242046 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:13.242068 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:13.991347 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:13.991402 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:14.459168 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:14.459193 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:14.459198 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:14.991649 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000074
2015-04-10 09:07:14.991700 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:15.465221 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:15.465245 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:15.465250 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:16.469815 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:16.469841 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:16.469846 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:16.469895 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:16.469917 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:17.478074 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:17.478111 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:17.478120 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:18.479443 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:18.479519 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:18.479522 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:19.491324 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:19.491333 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:19.491336 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:19.991776 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000076
2015-04-10 09:07:19.991827 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:20.491418 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:20.491437 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:20.491444 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:20.741349 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:20.741380 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:21.491522 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:21.491546 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:21.491551 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:21.844481 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:21.844521 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:22.491632 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:22.491656 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:22.491660 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:23.491751 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:23.491777 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:23.491780 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:23.544891 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:23.544943 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:24.491871 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:24.491904 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:24.491910 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:24.991906 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000078
2015-04-10 09:07:24.991958 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:25.492030 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:25.492067 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:25.492080 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:26.492158 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:26.492187 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:26.492194 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:27.492273 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:27.492293 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:27.492298 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:28.741391 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:28.741418 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:28.741424 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:29.491435 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:29.491472 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:29.741501 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:29.741530 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:29.741534 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:29.992033 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000075
2015-04-10 09:07:29.992087 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:30.741610 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:30.741633 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:30.741636 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:31.741722 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:31.741755 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:31.741760 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:31.991535 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:31.991572 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:32.741851 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:32.741878 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:32.741884 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:33.241436 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:33.241472 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:33.741999 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:33.742027 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:33.742033 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:34.742111 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:34.742135 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:34.742143 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:34.992161 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000074
2015-04-10 09:07:34.992200 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:35.742224 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:35.742264 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:35.742270 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:36.742343 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:36.742366 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:36.742371 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:37.742443 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:37.742468 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:37.742474 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:38.742765 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:38.742855 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:38.742860 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:39.241535 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:39.241576 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:39.743091 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:39.743119 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:39.743124 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:39.992230 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000029
2015-04-10 09:07:39.992285 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:40.743250 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:40.743275 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:40.743280 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:41.743452 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:41.743479 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:41.743489 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:42.241485 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:42.241513 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:42.743663 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:42.743690 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:42.743697 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:43.743810 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:43.743844 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:43.743850 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:44.743929 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:44.743952 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:44.743955 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:44.992370 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000083
2015-04-10 09:07:44.992434 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:45.744173 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:45.744209 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:45.744216 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:46.744428 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:46.744453 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:46.744458 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:46.991467 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:46.991498 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:47.744564 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:47.744591 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:47.744597 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:48.744670 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:48.744688 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:48.744693 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:49.744765 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:49.744787 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:49.744792 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:49.992518 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000083
2015-04-10 09:07:49.992558 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:50.744927 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:50.744954 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:50.744959 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:51.991591 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:51.991621 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:51.991626 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:52.491588 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:52.491612 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:52.991701 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:52.991724 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:52.991728 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:53.991798 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:53.991817 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:53.991821 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:54.991905 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:54.991935 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:54.991940 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:54.992632 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000072
2015-04-10 09:07:54.992674 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:07:55.491831 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:55.491855 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:55.992018 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:55.992043 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:55.992049 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:56.992146 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:56.992183 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:56.992188 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:57.746103 7f5689cdf700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:57.746120 7f5689cdf700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:57.746122 7f5689cdf700 20 osd.15 3814 _dispatch 0x1a456c0 osd_map(3814..3814 src has 2527..3814) v3
2015-04-10 09:07:57.746586 7f5689cdf700 3 osd.15 3814 handle_osd_map epochs [3814,3814], i have 3814, src has [2527,3814]
2015-04-10 09:07:57.746598 7f5689cdf700 10 osd.15 3814 no new maps here, dropping
2015-04-10 09:07:57.746600 7f5689cdf700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:57.746602 7f5689cdf700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:57.992285 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:57.992325 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:57.992330 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:58.992014 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:58.992048 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:58.992406 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:58.992434 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:58.992439 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:59.492192 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:59.492220 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:07:59.992518 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:07:59.992543 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:07:59.992547 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:07:59.992749 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000074
2015-04-10 09:07:59.992788 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:00.992636 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:00.992662 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:00.992667 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:01.991610 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:01.991643 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:01.992743 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:01.992783 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:01.992788 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:02.992862 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:02.992897 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:02.992902 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:03.992988 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:03.993034 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:03.993039 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:05.059679 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.066888
2015-04-10 09:08:05.059733 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:05.059939 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:05.059980 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:05.059984 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:06.241609 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:06.241636 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:06.241642 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:06.241653 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:06.241673 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:07.241718 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:07.241742 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:07.241750 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:07.352657 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:07.352687 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:07.991627 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:07.991667 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:08.241825 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:08.241856 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:08.241866 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:09.241941 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:09.241965 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:09.241973 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:10.241632 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.181897
2015-04-10 09:08:10.241679 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:10.242050 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:10.242077 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:10.242081 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:11.242336 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:11.242429 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:11.242432 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:12.241645 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:12.241667 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:12.242509 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:12.242568 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:12.242574 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:13.242646 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:13.242665 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:13.242670 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:14.242744 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:14.242765 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:14.242772 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:14.741700 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:14.741735 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:15.241766 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000085
2015-04-10 09:08:15.241806 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:15.242849 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:15.242870 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:15.242873 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:16.242952 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:16.242970 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:16.242973 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:16.441896 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:16.441926 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:16.991719 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:16.991759 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:17.243048 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:17.243078 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:17.243082 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:18.389865 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:18.389893 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:18.389898 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:19.491649 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:19.491671 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:19.491677 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:20.241895 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000088
2015-04-10 09:08:20.241938 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:20.491750 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:20.491778 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:20.491784 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:21.241652 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:21.241680 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:21.491847 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:21.491876 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:21.491881 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:22.491715 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:22.491744 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:22.491964 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:22.491991 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:22.491996 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:23.492092 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:23.492128 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:23.492136 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:24.492221 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:24.492259 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:24.492265 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:25.242020 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000080
2015-04-10 09:08:25.242069 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:25.492338 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:25.492365 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:25.492375 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:25.991872 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:25.991898 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:26.492447 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:26.492471 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:26.492475 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:27.492548 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:27.492574 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:27.492583 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:28.492657 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:28.492695 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:28.492699 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:29.492067 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:29.492099 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:29.492745 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:29.492771 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:29.492778 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:30.242145 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000075
2015-04-10 09:08:30.242180 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:30.492851 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:30.492872 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:30.492876 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:31.492955 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:31.492988 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:31.492991 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:32.491800 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:32.491837 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:32.493066 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:32.493093 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:32.493097 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:33.493273 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:33.493296 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:33.493300 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:34.507001 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:34.507025 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:34.507030 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:35.242305 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry woke after 5.000124
2015-04-10 09:08:35.242346 7f568fc33700 20 filestore(/var/lib/ceph/osd/osd.15) sync_entry waiting for max_interval 5.000000
2015-04-10 09:08:35.507738 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:35.507764 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:35.507768 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:36.507857 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:36.507888 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:36.507894 7f5692c39700 10 osd.15 3814 do_waiters -- finish
2015-04-10 09:08:37.192005 7f5680ccd700 20 osd.15 3814 update_osd_stat osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:37.192059 7f5680ccd700 5 osd.15 3814 heartbeat: osd_stat(85104 kB used, 930 GB avail, 931 GB total, peers []/[] op hist [])
2015-04-10 09:08:37.741760 7f5692c39700 5 osd.15 3814 tick
2015-04-10 09:08:37.741774 7f5692c39700 10 osd.15 3814 do_waiters -- start
2015-04-10 09:08:37.741776 7f5692c39700 10 osd.15 3814 do_waiters -- finish
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
