Re: ceph monitor is very slow

Linux Chips <linux.chips@xxxxxxxxx> · Sat, 23 May 2015 03:40:05 +0300



    the other monitor shows the folowing in
      the logs:

      
      2015-05-23 03:35:05.425037 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:35:20.382219 7fef908d9700  0
      mon.monitor02@1(probing).data_health(682) update_stats avail 87%
      total 146 GB, used 11729 MB, avail 127 GB

      2015-05-23 03:35:34.417562 7fef8f758700  1
      mon.monitor02@1(probing).data_health(682) service_dispatch not in
      quorum -- drop message

      2015-05-23 03:35:34.423806 7fef8f758700  0 log_channel(cluster)
      log [INF] : mon.monitor02 calling new monitor election

      2015-05-23 03:35:34.423922 7fef8f758700  1
      mon.monitor02@1(electing).elector(682) init, last seen epoch 682

      2015-05-23 03:35:35.169595 7fefe8529700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x67c7c80 sd=42 :6789 s=0 pgs=0 cs=0 l=0 c=0x6734840).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34064/0)

      2015-05-23 03:35:44.573823 7fef908d9700  1
      mon.monitor02@1(peon).paxos(paxos updating c 54355808..54356341)
      lease_timeout -- calling new election

      2015-05-23 03:35:45.185607 7fefe8529700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x6667340 sd=42 :6789 s=0 pgs=0 cs=0 l=0 c=0x6300f20).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34065/0)

      2015-05-23 03:35:45.488615 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:35:55.202086 7ff0051d5700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x77d4ec0 sd=159 :6789 s=0 pgs=0 cs=0 l=0 c=0x822c520).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34066/0)

      2015-05-23 03:35:55.504464 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:36:05.217264 7fef74ef1700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x748f9c0 sd=33 :6789 s=0 pgs=0 cs=0 l=0 c=0x7200520).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34067/0)

      2015-05-23 03:36:05.520334 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:36:20.388516 7fef908d9700  0
      mon.monitor02@1(probing).data_health(684) update_stats avail 87%
      total 146 GB, used 11665 MB, avail 127 GB

      2015-05-23 03:36:35.266879 7fefe7df2700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x77d38c0 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x822db20).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34070/0)

      2015-05-23 03:36:35.568066 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:37:11.433662 7fef8f758700  1
      mon.monitor02@1(probing).data_health(684) service_dispatch not in
      quorum -- drop message

      2015-05-23 03:37:11.449583 7fef8f758700  0 log_channel(cluster)
      log [INF] : mon.monitor02 calling new monitor election

      2015-05-23 03:37:11.449771 7fef8f758700  1
      mon.monitor02@1(electing).elector(684) init, last seen epoch 684

      2015-05-23 03:37:20.394614 7fef908d9700  0
      mon.monitor02@1(peon).data_health(686) update_stats avail 87%
      total 146 GB, used 11601 MB, avail 127 GB

      2015-05-23 03:37:21.605784 7fef908d9700  1
      mon.monitor02@1(peon).paxos(paxos updating c 54355808..54356342)
      lease_timeout -- calling new election

      2015-05-23 03:37:25.349501 7ff001c46700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x7462100 sd=68 :6789 s=0 pgs=0 cs=0 l=0 c=0x822d9c0).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34075/0)

      2015-05-23 03:37:25.647560 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:37:45.382132 7fef7fbdf700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x7e563c0 sd=161 :6789 s=0 pgs=0 cs=0 l=0 c=0x6742260).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34077/0)

      2015-05-23 03:37:45.679437 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:37:55.398536 7feff87b7700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x7c89080 sd=87 :6789 s=0 pgs=0 cs=0 l=0 c=0x7475340).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34078/0)

      2015-05-23 03:37:55.695356 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:38:20.400672 7fef908d9700  0
      mon.monitor02@1(probing).data_health(686) update_stats avail 87%
      total 146 GB, used 11601 MB, avail 127 GB

      2015-05-23 03:38:25.446655 7feffd697700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x67c4b00 sd=56 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fcbc80).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34081/0)

      2015-05-23 03:38:25.743176 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:38:48.682917 7fef8f758700  1
      mon.monitor02@1(probing).data_health(686) service_dispatch not in
      quorum -- drop message

      2015-05-23 03:38:48.690102 7fef8f758700  0 log_channel(cluster)
      log [INF] : mon.monitor02 calling new monitor election

      2015-05-23 03:38:48.690237 7fef8f758700  1
      mon.monitor02@1(electing).elector(686) init, last seen epoch 686

      2015-05-23 03:38:58.868484 7fef908d9700  1
      mon.monitor02@1(peon).paxos(paxos updating c 54355808..54356343)
      lease_timeout -- calling new election

      2015-05-23 03:39:15.523290 7fefb6146700  0 --
      192.168.217.203:6789/0 >> 192.168.1.69:0/1293654400
      pipe(0x778f9c0 sd=104 :6789 s=0 pgs=0 cs=0 l=0 c=0x6f50dc0).accept
      peer addr is really 192.168.1.69:0/1293654400 (socket is
      192.168.1.69:34086/0)

      2015-05-23 03:39:15.822816 7fef8f758700  1
      mon.monitor02@1(probing) e6 _ms_dispatch dropping stray message
      mon_subscribe({monmap=0+,osdmap=190533}) from client.10756902
      192.168.1.69:0/1293654400

      2015-05-23 03:39:20.406852 7fef908d9700  0
      mon.monitor02@1(probing).data_health(688) update_stats avail 87%
      total 146 GB, used 11559 MB, avail 127 GB

      
      On 05/23/2015 03:14 AM, Linux Chips wrote:

    
      Dears,

      i have a cluster of ceph, with two monitors.

      earlier i tried to add a monitor but it stuck syncing and refused
      to join the quorum.

      then the two monitors i had got there stores very big, ~25GB.

      i restarted one of the monitors (a suggestion to get the mon to
      clear its previous unwonted maps, i can not find it in my
      browser's history now :D ) and it then kept getting in and out of
      the quorum.

      i found that it is slow in response to any thing even to try to
      get the mon_state. the bottle neck was the disk.

      the monitor is reading at full speed from the disk, continuously,
      at a rate ~80 MB/s

      now i was trying to add reliability to the cluster and ended up
      with a broken one.

      the cluster is a 15 osd node and 2 mons, no mds.

      any idea on why the mon is reading this much of a data? what can i
      do to debug it?

      i tried setting the debug level higher. i only get things like
      these:

      
      2015-05-23 03:10:40.289533 7f74b0ada700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.106:0/3041278 client protocol
      0

      2015-05-23 03:10:40.308461 7f74b03d3700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.101:0/1006517 client protocol
      0

      2015-05-23 03:10:40.308704 7f74b00d0700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.101:0/4006517 client protocol
      0

      2015-05-23 03:10:40.308771 7f74b01d1700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.101:0/2006517 client protocol
      0

      2015-05-23 03:10:40.308792 7f74affcf700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.101:0/3006517 client protocol
      0

      2015-05-23 03:10:40.309587 7f74aeebe700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.101:0/1007420 client protocol
      0

      2015-05-23 03:10:40.348127 7f74aeaba700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.101:0/1020905 client protocol
      0

      2015-05-23 03:10:40.351846 7f74ad6a6700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.106:0/2011199 client protocol
      0

      2015-05-23 03:10:40.365362 7f74acc9c700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.103:0/1023531 client protocol
      0

      
      and then things like 

      
      2015-05-23 03:12:40.579872 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.579879 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.579883 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0x252ecec0 next
      190533 (onetime)

      2015-05-23 03:12:40.579923 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.579928 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.579932 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0x9426b40 next
      190533 (onetime)

      2015-05-23 03:12:40.579965 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.579970 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.579974 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0x359d8100 next
      190533 (onetime)

      2015-05-23 03:12:40.580010 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.580016 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.580019 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0xfa99240 next
      190533 (onetime)

      2015-05-23 03:12:40.580053 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.580058 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.580061 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0x1ac933c0 next
      190533 (onetime)

      2015-05-23 03:12:40.580094 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.580099 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.580102 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0xa538000 next
      190533 (onetime)

      2015-05-23 03:12:40.580135 7f74b7f32700 10
      mon.monitor01@0(electing) e6 handle_subscribe
      mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.580140 7f74b7f32700 10
      mon.monitor01@0(electing) e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.580143 7f74b7f32700 10
      mon.monitor01@0(electing).osd e190532 check_sub 0x1ac93140 next
      190533 (onetime)

      2015-05-23 03:12:40.581487 7f74b7f32700  5
      mon.monitor01@0(electing).elector(655) handle_ack from mon.1

      2015-05-23 03:12:40.581492 7f74b7f32700  5
      mon.monitor01@0(electing).elector(655)  so far i have
      {0=70368744177663,1=70368744177663}

      2015-05-23 03:12:40.581497 7f74b7f32700 10
      mon.monitor01@0(electing).elector(655) bump_epoch 655 to 656

      2015-05-23 03:12:40.603273 7f74b7f32700 10
      mon.monitor01@0(electing) e6 join_election

      2015-05-23 03:12:40.603281 7f74b7f32700 10
      mon.monitor01@0(electing) e6 _reset

      2015-05-23 03:12:40.603283 7f74b7f32700 10
      mon.monitor01@0(electing) e6 cancel_probe_timeout (none scheduled)

      2015-05-23 03:12:40.603285 7f74b7f32700 10
      mon.monitor01@0(electing) e6 timecheck_finish

      2015-05-23 03:12:40.603286 7f74b7f32700 10
      mon.monitor01@0(electing) e6 scrub_reset

      2015-05-23 03:12:40.603315 7f74b7f32700 10
      mon.monitor01@0(electing) e6 win_election epoch 656 quorum 0,1
      features 70368744177663

      2015-05-23 03:12:40.603326 7f74b7f32700  0 log_channel(cluster)
      log [INF] : mon.monitor01@0 won leader election with quorum 0,1

      2015-05-23 03:12:40.615572 7f74a9262700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.103:0/1025658 client protocol
      0

      2015-05-23 03:12:40.657969 7f74a9060700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.103:0/2023531 client protocol
      0

      2015-05-23 03:12:40.681508 7f74a8e5e700 10 mon.monitor01@0(leader)
      e6 ms_verify_authorizer 192.168.213.103:0/1011519 client protocol
      0

      2015-05-23 03:12:40.686338 7f74b7f32700 10
      mon.monitor01@0(leader).data_health(656) start_epoch epoch 656

      2015-05-23 03:12:40.686352 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck_finish

      2015-05-23 03:12:40.686357 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686360 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17275596
      192.168.213.103:0/2004855 is open for client.17275596
      192.168.213.103:0/2004855

      2015-05-23 03:12:40.686371 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686402 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686405 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17272524
      192.168.213.103:0/2012444 is open for client.17272524
      192.168.213.103:0/2012444

      2015-05-23 03:12:40.686413 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686435 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686438 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.18215582
      192.168.213.106:0/4041278 is open for client.18215582
      192.168.213.106:0/4041278

      2015-05-23 03:12:40.686447 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686496 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686500 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17273120
      192.168.213.103:0/1019238 is open for client.17273120
      192.168.213.103:0/1019238

      2015-05-23 03:12:40.686509 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686535 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686536 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17272886
      192.168.213.103:0/2015460 is open for client.17272886
      192.168.213.103:0/2015460

      2015-05-23 03:12:40.686540 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686550 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686552 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17272970
      192.168.213.103:0/1017082 is open for client.17272970
      192.168.213.103:0/1017082

      2015-05-23 03:12:40.686567 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686577 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686578 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.18216138
      192.168.213.103:0/3038561 is open for client.18216138
      192.168.213.103:0/3038561

      2015-05-23 03:12:40.686582 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686592 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686593 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17272520
      192.168.213.103:0/1012444 is open for client.17272520
      192.168.213.103:0/1012444

      2015-05-23 03:12:40.686597 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686608 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686610 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.19621431
      192.168.213.103:0/2025791 is open for client.19621431
      192.168.213.103:0/2025791

      2015-05-23 03:12:40.686613 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686623 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686624 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17273004
      192.168.213.103:0/1017826 is open for client.17273004
      192.168.213.103:0/1017826

      2015-05-23 03:12:40.686628 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686638 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686639 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17436946
      192.168.213.101:0/2040116 is open for client.17436946
      192.168.213.101:0/2040116

      2015-05-23 03:12:40.686643 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686653 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686654 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17275494
      192.168.213.103:0/1002572 is open for client.17275494
      192.168.213.103:0/1002572

      2015-05-23 03:12:40.686658 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686668 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686669 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.18216140
      192.168.213.103:0/4038561 is open for client.18216140
      192.168.213.103:0/4038561

      2015-05-23 03:12:40.686673 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686683 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686684 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17272872
      192.168.213.103:0/1015258 is open for client.17272872
      192.168.213.103:0/1015258

      2015-05-23 03:12:40.686688 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686697 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686699 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17275544
      192.168.213.101:0/2033630 is open for client.17275544
      192.168.213.101:0/2033630

      2015-05-23 03:12:40.686703 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.686723 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.686725 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17275542
      192.168.213.101:0/1033630 is open for client.17275542
      192.168.213.101:0/1033630

      
      ...

      ...

      ...

      
      2015-05-23 03:12:40.687360 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.687370 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.687372 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17274530
      192.168.213.103:0/1031546 is open for client.17274530
      192.168.213.103:0/1031546

      2015-05-23 03:12:40.687376 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.687387 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 resend_routed_requests

      2015-05-23 03:12:40.687389 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 register_cluster_logger - already registered

      2015-05-23 03:12:40.687391 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck_start

      2015-05-23 03:12:40.687392 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck_start_round curr 0

      2015-05-23 03:12:40.687394 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck_start_round new 1

      2015-05-23 03:12:40.687395 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck

      2015-05-23 03:12:40.687396 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck start timecheck epoch 656 round 1

      2015-05-23 03:12:40.687402 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck send time_check( ping e 656 r 1 ) v1 to mon.1
      192.168.217.203:6789/0

      2015-05-23 03:12:40.687413 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 timecheck_start_round setting up next event

      2015-05-23 03:12:40.690522 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.690524 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17273862
      192.168.213.103:0/1025658 is open for client.17273862
      192.168.213.103:0/1025658

      2015-05-23 03:12:40.690528 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.690545 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.690547 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17273598
      192.168.213.103:0/2023531 is open for client.17273598
      192.168.213.103:0/2023531

      2015-05-23 03:12:40.690550 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.690566 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 do not have session, making new one

      2015-05-23 03:12:40.690568 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 ms_dispatch new session MonSession: client.17272358
      192.168.213.103:0/1011519 is open for client.17272358
      192.168.213.103:0/1011519

      2015-05-23 03:12:40.690571 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 setting timeout on session

      2015-05-23 03:12:40.690590 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690593 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690595 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0x252ecec0 next
      190533 (onetime)

      2015-05-23 03:12:40.690615 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690618 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690620 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0x9426b40 next
      190533 (onetime)

      2015-05-23 03:12:40.690637 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690640 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690642 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0x359d8100 next
      190533 (onetime)

      2015-05-23 03:12:40.690659 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690662 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690663 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0xfa99240 next
      190533 (onetime)

      2015-05-23 03:12:40.690681 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690683 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690685 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0x1ac933c0 next
      190533 (onetime)

      2015-05-23 03:12:40.690701 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690704 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690705 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0xa538000 next
      190533 (onetime)

      2015-05-23 03:12:40.690724 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690727 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690729 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0x1ac93140 next
      190533 (onetime)

      2015-05-23 03:12:40.690746 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 handle_subscribe mon_subscribe({monmap=7+,osdmap=190533}) v2

      2015-05-23 03:12:40.690749 7f74b7f32700 10 mon.monitor01@0(leader)
      e6 check_sub monmap next 7 have 6

      2015-05-23 03:12:40.690750 7f74b7f32700 10
      mon.monitor01@0(leader).osd e190532 check_sub 0x252ecec0 next
      190533 (onetime)

      
      any clue on why i am facing this issue?

      any help is appreciated.

      thanks

      
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com