Re: Upgrade from Giant to Hammer and after some basic operations most of the OSD's went down

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Here it is

Br,
Tuomas


-----Original Message-----
From: Sage Weil [mailto:sweil@xxxxxxxxxx] 
Sent: 28. huhtikuuta 2015 21:57
To: Tuomas Juntunen
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: RE:  Upgrade from Giant to Hammer and after some basic
operations most of the OSD's went down

On Tue, 28 Apr 2015, Tuomas Juntunen wrote:
> Hi
> 
> Here's the file.

Hrm.  Can you also attach

 ceph osd dump -f json-pretty 17846

> The cluster was working totally fine for months before this, no 
> problems at all.
> 
> 'ceph osd tree' still shows 11 osd's up, even though all of them are down.
> If I add a new one, that will get added to the list and it goes up, 
> but when I stop it, the status will still show it as up.
> 
> Here's a log for that osd http://beta.xaasbox.com/ceph/ceph-osd.37.log

Does your ceph.conf have any changes to osd_heartbeat_* configurables?  
It is very strange that the OSDs aren't getting marked down.

sage

> 
> Br,
> Tuomas
> 
> 
> -----Original Message-----
> From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> Sent: 28. huhtikuuta 2015 20:02
> To: Tuomas Juntunen
> Cc: ceph-users@xxxxxxxxxxxxxx
> Subject: RE:  Upgrade from Giant to Hammer and after some 
> basic operations most of the OSD's went down
> 
> On Tue, 28 Apr 2015, Tuomas Juntunen wrote:
> > Just to add some more interesting behavior to my problem, is that 
> > monitors are not updating the status of OSD's.
> 
> Yeah, this is very strange.  I see
> 
> 2015-04-27 22:25:26.142245 7f78d793a900 10 osd.15 17882 _get_pool 4 
> cached_removed_snaps [1~1,4~a,f~2,12~10,23~4,28~2,2e~2,32~c,3f~4]
> 
> but pool 4 (images) has no snaps and a zeroed snap_seq in the osdmap 
> dump you provided.  Can you attach
> 
>  ceph osd dump -f json-pretty 17882
> 
> ?
> 
> The fact that your mon status isn't changing makes me wonder if these 
> are two different ceph clusters or something similarly silly...  if 
> the mds daemons are down the status should show stale.  If all osds 
> stop you might not see the osds go down for ~10 minutes (since they 
> aren't monitoring each other).
> 
> sage
> 
> > 
> > Even when I stop all the remaining OSD's, ceph osd tree shows them as
up.
> > Also there the status of mons and mds doesn't seem to update 
> > correctly in my opinion.
> > 
> > Below is a copy of status when one mon and mds are stopped and all 
> > of the osd's are also stopped.
> > 
> >      monmap e7: 3 mons at
> >
{ceph1=10.20.0.11:6789/0,ceph2=10.20.0.12:6789/0,ceph3=10.20.0.13:6789/0}
> >             election epoch 48, quorum 0,1 ceph1,ceph2
> >      mdsmap e1750: 1/1/1 up {0=ceph2=up:replay}, 1 up:standby
> >      osdmap e18132: 37 osds: 11 up, 11 in
> > 
> > Br,
> > Tuomas
> > 
> > 
> > -----Original Message-----
> > From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> > Sent: 27. huhtikuuta 2015 22:22
> > To: Tuomas Juntunen
> > Cc: ceph-users@xxxxxxxxxxxxxx
> > Subject: RE:  Upgrade from Giant to Hammer and after 
> > some basic operations most of the OSD's went down
> > 
> > On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> > > Hey
> > > 
> > > Got the log, you can get it from
> > > http://beta.xaasbox.com/ceph/ceph-osd.15.log
> > 
> > Can you repeat this with 'debug osd = 20'?  Thanks!
> > 
> > sage
> > 
> > > 
> > > Br,
> > > Tuomas
> > > 
> > > 
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> > > Sent: 27. huhtikuuta 2015 20:45
> > > To: Tuomas Juntunen
> > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > Subject: Re:  Upgrade from Giant to Hammer and after 
> > > some basic operations most of the OSD's went down
> > > 
> > > Yeah, no snaps:
> > > 
> > > images:
> > >             "snap_mode": "selfmanaged",
> > >             "snap_seq": 0,
> > >             "snap_epoch": 17882,
> > >             "pool_snaps": [],
> > >             "removed_snaps": "[]",
> > > 
> > > img:
> > >             "snap_mode": "selfmanaged",
> > >             "snap_seq": 0,
> > >             "snap_epoch": 0,
> > >             "pool_snaps": [],
> > >             "removed_snaps": "[]",
> > > 
> > > ...and actually the log shows this happens on pool 2 (rbd), which 
> > > has
> > > 
> > >             "snap_mode": "selfmanaged",
> > >             "snap_seq": 0,
> > >             "snap_epoch": 0,
> > >             "pool_snaps": [],
> > >             "removed_snaps": "[]",
> > > 
> > > I'm guessin gthe offending code is
> > > 
> > >     pi->build_removed_snaps(newly_removed_snaps);
> > >     newly_removed_snaps.subtract(cached_removed_snaps);
> > > 
> > > so newly_removed_snaps should be empty, and apparently 
> > > cached_removed_snaps is not?  Maybe one of your older osdmaps has 
> > > snap info for rbd?  It doesn't make sense.  :/  Maybe
> > > 
> > >  ceph osd dump 18127 -f json-pretty
> > > 
> > > just to be certain?  I've pushed a branch 'wip-hammer-snaps' that 
> > > will appear at gitbuilder.ceph.com in 20-30 minutes that will 
> > > output some additional debug info.  It will be at
> > > 
> > > 	
> > > http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/wip-ha
> > > mm
> > > er
> > > -sanps
> > > 
> > > or similar, depending on your distro.  Can you install it one on 
> > > node and start and osd with logging to reproduce the crash?
> > > 
> > > Thanks!
> > > sage
> > > 
> > > 
> > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> > > 
> > > > Hi
> > > > 
> > > > Here you go
> > > > 
> > > > Br,
> > > > Tuomas
> > > > 
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> > > > Sent: 27. huhtikuuta 2015 19:23
> > > > To: Tuomas Juntunen
> > > > Cc: 'Samuel Just'; ceph-users@xxxxxxxxxxxxxx
> > > > Subject: Re:  Upgrade from Giant to Hammer and after 
> > > > some basic operations most of the OSD's went down
> > > > 
> > > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> > > > > Thanks for the info.
> > > > > 
> > > > > For my knowledge there was no snapshots on that pool, but 
> > > > > cannot verify that.
> > > > 
> > > > Can you attach a 'ceph osd dump -f json-pretty'?  That will shed 
> > > > a bit more light on what happened (and the simplest way to fix it).
> > > > 
> > > > sage
> > > > 
> > > > 
> > > > > Any way to make this work again? Removing the tier and other 
> > > > > settings didn't fix it, I tried it the second this happened.
> > > > > 
> > > > > Br,
> > > > > Tuomas
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Samuel Just [mailto:sjust@xxxxxxxxxx]
> > > > > Sent: 27. huhtikuuta 2015 15:50
> > > > > To: tuomas juntunen
> > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > Subject: Re:  Upgrade from Giant to Hammer and 
> > > > > after some basic operations most of the OSD's went down
> > > > > 
> > > > > So, the base tier is what determines the snapshots for the 
> > > > > cache/base pool
> > > > amalgam.  You added a populated pool complete with snapshots on 
> > > > top of a base tier without snapshots.  Apparently, it caused an 
> > > > existential crisis for the snapshot code.  That's one of the 
> > > > reasons why there is a --force-nonempty flag for that operation, 
> > > > I think.  I think the immediate answer is probably to disallow 
> > > > pools with snapshots as a cache tier altogether until we think 
> > > > of a good way to
> > make it work.
> > > > > -Sam
> > > > > 
> > > > > ----- Original Message -----
> > > > > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > > > > To: "Samuel Just" <sjust@xxxxxxxxxx>
> > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > Sent: Monday, April 27, 2015 4:56:58 AM
> > > > > Subject: Re:  Upgrade from Giant to Hammer and 
> > > > > after some basic operations most of the OSD's went down
> > > > > 
> > > > > 
> > > > > 
> > > > > The following:
> > > > > 
> > > > > ceph osd tier add img images --force-nonempty ceph osd tier 
> > > > > cache-mode images forward ceph osd tier set-overlay img images
> > > > > 
> > > > > Idea was to make images as a tier to img, move data to img 
> > > > > then change
> > > > clients to use the new img pool.
> > > > > 
> > > > > Br,
> > > > > Tuomas
> > > > > 
> > > > > > Can you explain exactly what you mean by:
> > > > > >
> > > > > > "Also I created one pool for tier to be able to move data 
> > > > > > without
> > > > outage."
> > > > > >
> > > > > > -Sam
> > > > > > ----- Original Message -----
> > > > > > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > > > > > To: "Ian Colle" <icolle@xxxxxxxxxx>
> > > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > > Sent: Monday, April 27, 2015 4:23:44 AM
> > > > > > Subject: Re:  Upgrade from Giant to Hammer and 
> > > > > > after some basic operations most of the OSD's went down
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > Any solution for this yet?
> > > > > >
> > > > > > Br,
> > > > > > Tuomas
> > > > > >
> > > > > >> It looks like you may have hit
> > > > > >> http://tracker.ceph.com/issues/7915
> > > > > >>
> > > > > >> Ian R. Colle
> > > > > >> Global Director
> > > > > >> of Software Engineering
> > > > > >> Red Hat (Inktank is now part of Red Hat!) 
> > > > > >> http://www.linkedin.com/in/ircolle
> > > > > >> http://www.twitter.com/ircolle
> > > > > >> Cell: +1.303.601.7713
> > > > > >> Email: icolle@xxxxxxxxxx
> > > > > >>
> > > > > >> ----- Original Message -----
> > > > > >> From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > > > > >> To: ceph-users@xxxxxxxxxxxxxx
> > > > > >> Sent: Monday, April 27, 2015 1:56:29 PM
> > > > > >> Subject:  Upgrade from Giant to Hammer and 
> > > > > >> after some basic operations most of the OSD's went down
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> I upgraded Ceph from 0.87 Giant to 0.94.1 Hammer
> > > > > >>
> > > > > >> Then created new pools and deleted some old ones. Also I 
> > > > > >> created one pool for tier to be able to move data without
outage.
> > > > > >>
> > > > > >> After these operations all but 10 OSD's are down and 
> > > > > >> creating this kind of messages to logs, I get more than 
> > > > > >> 100gb of these in a
> > > night:
> > > > > >>
> > > > > >>  -19> 2015-04-27 10:17:08.808584 7fd8e748d700  5 osd.23
pg_epoch:
> 
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] enter Started
> > > > > >>    -18> 2015-04-27 10:17:08.808596 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] enter Start
> > > > > >>    -17> 2015-04-27 10:17:08.808608 7fd8e748d700  1 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
> > > > > >>    -16> 2015-04-27 10:17:08.808621 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] exit Start 0.000025 0 0.000000
> > > > > >>    -15> 2015-04-27 10:17:08.808637 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] enter Started/Stray
> > > > > >>    -14> 2015-04-27 10:17:08.808796 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive 
> > > > > >> NOTIFY] exit Reset 0.119467 4 0.000037
> > > > > >>    -13> 2015-04-27 10:17:08.808817 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive 
> > > > > >> NOTIFY] enter Started
> > > > > >>    -12> 2015-04-27 10:17:08.808828 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive 
> > > > > >> NOTIFY] enter Start
> > > > > >>    -11> 2015-04-27 10:17:08.808838 7fd8e748d700  1 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive 
> > > > > >> NOTIFY]
> > > > > >> state<Start>: transitioning to Stray
> > > > > >>    -10> 2015-04-27 10:17:08.808849 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive 
> > > > > >> NOTIFY] exit Start 0.000020 0 0.000000
> > > > > >>     -9> 2015-04-27 10:17:08.808861 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive 
> > > > > >> NOTIFY] enter Started/Stray
> > > > > >>     -8> 2015-04-27 10:17:08.809427 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive] exit Reset 7.511623 45 0.000165
> > > > > >>     -7> 2015-04-27 10:17:08.809445 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive] enter Started
> > > > > >>     -6> 2015-04-27 10:17:08.809456 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive] enter Start
> > > > > >>     -5> 2015-04-27 10:17:08.809468 7fd8e748d700  1 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive]
> > > > > >> state<Start>: transitioning to Primary
> > > > > >>     -4> 2015-04-27 10:17:08.809479 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive] exit Start 0.000023 0 0.000000
> > > > > >>     -3> 2015-04-27 10:17:08.809492 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive] enter Started/Primary
> > > > > >>     -2> 2015-04-27 10:17:08.809502 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> inactive] enter Started/Primary/Peering
> > > > > >>     -1> 2015-04-27 10:17:08.809513 7fd8e748d700  5 osd.23
> pg_epoch:
> > 
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 
> > > > > >> peering] enter Started/Primary/Peering/GetInfo
> > > > > >>      0> 2015-04-27 10:17:08.813837 7fd8e748d700 -1
> > > > ./include/interval_set.h:
> > > > > >> In
> > > > > >> function 'void interval_set<T>::erase(T, T) [with T =
snapid_t]' 
> > > > > >> thread
> > > > > >> 7fd8e748d700 time 2015-04-27 10:17:08.809899
> > > > > >> ./include/interval_set.h: 385: FAILED assert(_size >= 0)
> > > > > >>
> > > > > >>  ceph version 0.94.1
> > > > > >> (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
> > > > > >>  1: (ceph::__ceph_assert_fail(char const*, char const*, 
> > > > > >> int, char
> > > > > >> const*)+0x8b)
> > > > > >> [0xbc271b]
> > > > > >>  2: 
> > > > > >> (interval_set<snapid_t>::subtract(interval_set<snapid_t>
> > > > > >> const&)+0xb0) [0x82cd50]
> > > > > >>  3: (PGPool::update(std::tr1::shared_ptr<OSDMap
> > > > > >> const>)+0x52e) [0x80113e]
> > > > > >>  4: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap
> > > > > >> const>, std::tr1::shared_ptr<OSDMap const>, 
> > > > > >> const>std::vector<int,
> > > > > >> std::allocator<int> >&, int, std::vector<int, 
> > > > > >> std::allocator<int>
> > > > > >> >&, int, PG::RecoveryCtx*)+0x282) [0x801652]
> > > > > >>  5: (OSD::advance_pg(unsigned int, PG*, 
> > > > > >> ThreadPool::TPHandle&, PG::RecoveryCtx*, 
> > > > > >> std::set<boost::intrusive_ptr<PG>,
> > > > > >> std::less<boost::intrusive_ptr<PG> >, 
> > > > > >> std::allocator<boost::intrusive_ptr<PG> > >*)+0x2c3) 
> > > > > >> [0x6b0e43]
> > > > > >>  6: (OSD::process_peering_events(std::list<PG*,
> > > > > >> std::allocator<PG*>
> > > > > >> > const&,
> > > > > >> ThreadPool::TPHandle&)+0x21c) [0x6b191c]
> > > > > >>  7: (OSD::PeeringWQ::_process(std::list<PG*,
> > > > > >> std::allocator<PG*>
> > > > > >> > const&,
> > > > > >> ThreadPool::TPHandle&)+0x18) [0x709278]
> > > > > >>  8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e)
> > > > > >> [0xbb38ae]
> > > > > >>  9: (ThreadPool::WorkThread::entry()+0x10) [0xbb4950]
> > > > > >>  10: (()+0x8182) [0x7fd906946182]
> > > > > >>  11: (clone()+0x6d) [0x7fd904eb147d]
> > > > > >>
> > > > > >> Also by monitoring (ceph -w) I get the following messages, 
> > > > > >> also lots of
> > > > them.
> > > > > >>
> > > > > >> 2015-04-27 10:39:52.935812 mon.0 [INF] from='client.?
> > > > 10.20.0.13:0/1174409'
> > > > > >> entity='osd.30' cmd=[{"prefix": "osd crush create-or-move",
> "args":
> > > > > >> ["host=ceph3", "root=default"], "id": 30, "weight": 1.82}]: 
> > > > > >> dispatch
> > > > > >> 2015-04-27 10:39:53.297376 mon.0 [INF] from='client.?
> > > > 10.20.0.13:0/1174483'
> > > > > >> entity='osd.26' cmd=[{"prefix": "osd crush create-or-move",
> "args":
> > > > > >> ["host=ceph3", "root=default"], "id": 26, "weight": 1.82}]: 
> > > > > >> dispatch
> > > > > >>
> > > > > >>
> > > > > >> This is a cluster of 3 nodes with 36 OSD's, nodes are also 
> > > > > >> mons and mds's to save servers. All run Ubuntu 14.04.2.
> > > > > >>
> > > > > >> I have pretty much tried everything I could think of.
> > > > > >>
> > > > > >> Restarting daemons doesn't help.
> > > > > >>
> > > > > >> Any help would be appreciated. I can also provide more logs 
> > > > > >> if necessary. They just seem to get pretty large in few
moments.
> > > > > >>
> > > > > >> Thank you
> > > > > >> Tuomas
> > > > > >>
> > > > > >>
> > > > > >> _______________________________________________
> > > > > >> ceph-users mailing list
> > > > > >> ceph-users@xxxxxxxxxxxxxx
> > > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list
> > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > >
> > > > > >
> > > > > >
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > 
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > 
> > > > > 
> > > > 
> > > 
> > > 
> > 
> > 
> 
{
    "epoch": 17846,
    "fsid": "a2974742-3805-4cd3-bc79-765f2bddaefe",
    "created": "2014-10-15 20:43:45.186949",
    "modified": "2015-04-26 10:39:35.366678",
    "flags": "",
    "cluster_snapshot": "",
    "pool_max": 7,
    "max_osd": 71,
    "pools": [
        {
            "pool": 0,
            "pool_name": "data",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 45,
            "last_change": "1112",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 1,
            "pool_name": "metadata",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1114",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 2,
            "pool_name": "rbd",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 2,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1116",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 3,
            "pool_name": "volumes",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "9974",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 23,
            "snap_epoch": 9974,
            "pool_snaps": [],
            "removed_snaps": "[1~17]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 4,
            "pool_name": "images",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "17846",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 66,
            "snap_epoch": 17846,
            "pool_snaps": [],
            "removed_snaps": "[1~1,4~a,f~2,12~10,23~4,28~2,2e~2,32~c,3f~4]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 5,
            "pool_name": "backups",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1121",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 3,
            "snap_epoch": 705,
            "pool_snaps": [],
            "removed_snaps": "[1~3]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 6,
            "pool_name": "vms",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1122",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 7,
            "pool_name": "san",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "14096",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        }
    ],
    "osds": [
        {
            "osd": 0,
            "uuid": "757c3bc5-4d00-4344-8de4-82f5379c96af",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15738,
            "last_clean_end": 16079,
            "up_from": 16081,
            "up_thru": 17776,
            "down_at": 16079,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6981\/245797",
            "cluster_addr": "10.20.0.11:6881\/5245797",
            "heartbeat_back_addr": "10.20.0.11:6883\/5245797",
            "heartbeat_front_addr": "10.20.0.11:6887\/5245797",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 1,
            "uuid": "c7eaa4ac-99fc-46db-84aa-a67274896ec8",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15740,
            "last_clean_end": 16079,
            "up_from": 16083,
            "up_thru": 17781,
            "down_at": 16079,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6813\/245273",
            "cluster_addr": "10.20.0.11:6888\/4245273",
            "heartbeat_back_addr": "10.20.0.11:6890\/4245273",
            "heartbeat_front_addr": "10.20.0.11:6894\/4245273",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 2,
            "uuid": "206b2949-4adf-4789-8e06-f68a8ee819c9",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15739,
            "last_clean_end": 16079,
            "up_from": 16081,
            "up_thru": 17774,
            "down_at": 16079,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6808\/245269",
            "cluster_addr": "10.20.0.11:6811\/4245269",
            "heartbeat_back_addr": "10.20.0.11:6823\/4245269",
            "heartbeat_front_addr": "10.20.0.11:6831\/4245269",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 3,
            "uuid": "90b7c219-4dcd-48ea-a24d-f3b796a521e4",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15736,
            "last_clean_end": 16125,
            "up_from": 16128,
            "up_thru": 17774,
            "down_at": 16125,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6848\/245357",
            "cluster_addr": "10.20.0.11:6886\/6245357",
            "heartbeat_back_addr": "10.20.0.11:6891\/6245357",
            "heartbeat_front_addr": "10.20.0.11:6892\/6245357",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 4,
            "uuid": "049ef94f-121a-4e71-8ba6-27eaebf0a569",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15737,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17774,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6854\/245375",
            "cluster_addr": "10.20.0.11:6849\/5245375",
            "heartbeat_back_addr": "10.20.0.11:6850\/5245375",
            "heartbeat_front_addr": "10.20.0.11:6853\/5245375",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 5,
            "uuid": "2437a53b-339e-45af-b4de-0fc675d27405",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15734,
            "last_clean_end": 16087,
            "up_from": 16089,
            "up_thru": 17771,
            "down_at": 16087,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6834\/245323",
            "cluster_addr": "10.20.0.11:6803\/4245323",
            "heartbeat_back_addr": "10.20.0.11:6804\/4245323",
            "heartbeat_front_addr": "10.20.0.11:6805\/4245323",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 6,
            "uuid": "f117ceed-b1fd-4069-99fe-b7aba9f3ef8d",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15738,
            "last_clean_end": 16079,
            "up_from": 16083,
            "up_thru": 17778,
            "down_at": 16079,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6859\/245401",
            "cluster_addr": "10.20.0.11:6865\/5245401",
            "heartbeat_back_addr": "10.20.0.11:6868\/5245401",
            "heartbeat_front_addr": "10.20.0.11:6874\/5245401",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 7,
            "uuid": "e98e9b8a-9c62-4e3e-bdb4-c2c30103c0c1",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15730,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17771,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6801\/245246",
            "cluster_addr": "10.20.0.11:6812\/5245246",
            "heartbeat_back_addr": "10.20.0.11:6814\/5245246",
            "heartbeat_front_addr": "10.20.0.11:6821\/5245246",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 8,
            "uuid": "41e471cd-fafe-4422-8bf5-22018bbe1375",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15736,
            "last_clean_end": 17783,
            "up_from": 17795,
            "up_thru": 17799,
            "down_at": 17784,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6810\/458698",
            "cluster_addr": "10.20.0.11:6819\/458698",
            "heartbeat_back_addr": "10.20.0.11:6826\/458698",
            "heartbeat_front_addr": "10.20.0.11:6833\/458698",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 9,
            "uuid": "d68eeebd-d058-4b1c-a30a-994bf8fc8030",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15733,
            "last_clean_end": 16079,
            "up_from": 16081,
            "up_thru": 17733,
            "down_at": 16079,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6969\/245600",
            "cluster_addr": "10.20.0.11:6847\/5245600",
            "heartbeat_back_addr": "10.20.0.11:6860\/5245600",
            "heartbeat_front_addr": "10.20.0.11:6864\/5245600",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 10,
            "uuid": "660747d6-3f47-449a-bc69-5399b0d54ff6",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15736,
            "last_clean_end": 16124,
            "up_from": 16128,
            "up_thru": 17781,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6822\/245281",
            "cluster_addr": "10.20.0.11:6855\/4245281",
            "heartbeat_back_addr": "10.20.0.11:6856\/4245281",
            "heartbeat_front_addr": "10.20.0.11:6866\/4245281",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 11,
            "uuid": "805965b1-127f-44a6-9a05-8a643eb7a512",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15737,
            "last_clean_end": 17786,
            "up_from": 17801,
            "up_thru": 17804,
            "down_at": 17787,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6835\/458813",
            "cluster_addr": "10.20.0.11:6836\/458813",
            "heartbeat_back_addr": "10.20.0.11:6837\/458813",
            "heartbeat_front_addr": "10.20.0.11:6838\/458813",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 12,
            "uuid": "61fbfcbe-d642-478f-9620-f9d72ee96238",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16126,
            "up_from": 16129,
            "up_thru": 17818,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6873\/7542",
            "cluster_addr": "10.20.0.12:6947\/4007542",
            "heartbeat_back_addr": "10.20.0.12:6948\/4007542",
            "heartbeat_front_addr": "10.20.0.12:6949\/4007542",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 13,
            "uuid": "6faad33b-00be-42a4-92ba-08be5ab7f995",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16705,
            "up_from": 16707,
            "up_thru": 17829,
            "down_at": 16703,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6821\/7274",
            "cluster_addr": "10.20.0.12:6859\/5007274",
            "heartbeat_back_addr": "10.20.0.12:6860\/5007274",
            "heartbeat_front_addr": "10.20.0.12:6861\/5007274",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 14,
            "uuid": "f301705b-e725-443d-96e1-d9ec9aafe657",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16962,
            "up_from": 16966,
            "up_thru": 17801,
            "down_at": 16962,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6809\/7209",
            "cluster_addr": "10.20.0.12:6803\/7007209",
            "heartbeat_back_addr": "10.20.0.12:6804\/7007209",
            "heartbeat_front_addr": "10.20.0.12:6810\/7007209",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 15,
            "uuid": "536bf483-10de-44b0-8e1e-4f349fbe572a",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15785,
            "last_clean_end": 16700,
            "up_from": 16715,
            "up_thru": 17801,
            "down_at": 16693,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6881\/7613",
            "cluster_addr": "10.20.0.12:6845\/7007613",
            "heartbeat_back_addr": "10.20.0.12:6849\/7007613",
            "heartbeat_front_addr": "10.20.0.12:6850\/7007613",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 16,
            "uuid": "4185bd20-8eb0-4616-b36e-bacb181ae40e",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16688,
            "up_from": 16693,
            "up_thru": 17801,
            "down_at": 16686,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6813\/7210",
            "cluster_addr": "10.20.0.12:6814\/7007210",
            "heartbeat_back_addr": "10.20.0.12:6815\/7007210",
            "heartbeat_front_addr": "10.20.0.12:6816\/7007210",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 17,
            "uuid": "a6f2f5b4-477f-48f9-9acf-d5b7a6c88b98",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17803,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6817\/7215",
            "cluster_addr": "10.20.0.12:6840\/3007215",
            "heartbeat_back_addr": "10.20.0.12:6843\/3007215",
            "heartbeat_front_addr": "10.20.0.12:6844\/3007215",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 18,
            "uuid": "b31b0bd8-938a-496d-91bc-19bf4f794f82",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16700,
            "up_from": 16712,
            "up_thru": 17801,
            "down_at": 16700,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6805\/7185",
            "cluster_addr": "10.20.0.12:6829\/5007185",
            "heartbeat_back_addr": "10.20.0.12:6830\/5007185",
            "heartbeat_front_addr": "10.20.0.12:6841\/5007185",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 19,
            "uuid": "d76b6bd5-1ef3-436c-a75d-3587c515eb56",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16693,
            "up_from": 16706,
            "up_thru": 17801,
            "down_at": 16693,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6885\/7622",
            "cluster_addr": "10.20.0.12:6802\/7007622",
            "heartbeat_back_addr": "10.20.0.12:6818\/7007622",
            "heartbeat_front_addr": "10.20.0.12:6819\/7007622",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 20,
            "uuid": "8e4dd982-a4c5-4ca5-9fc5-243f55c4db57",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16703,
            "up_from": 16705,
            "up_thru": 17801,
            "down_at": 16703,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6825\/7316",
            "cluster_addr": "10.20.0.12:6827\/5007316",
            "heartbeat_back_addr": "10.20.0.12:6831\/5007316",
            "heartbeat_front_addr": "10.20.0.12:6851\/5007316",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 21,
            "uuid": "760aaf28-0a34-4bbc-af0c-2654b0a43fff",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16699,
            "up_from": 16701,
            "up_thru": 17801,
            "down_at": 16698,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6801\/7180",
            "cluster_addr": "10.20.0.12:6820\/7007180",
            "heartbeat_back_addr": "10.20.0.12:6822\/7007180",
            "heartbeat_front_addr": "10.20.0.12:6828\/7007180",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 22,
            "uuid": "40322a34-ab31-4760-b71e-a7672f812cb3",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16966,
            "up_from": 16969,
            "up_thru": 17801,
            "down_at": 16964,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6869\/7509",
            "cluster_addr": "10.20.0.12:6824\/8007509",
            "heartbeat_back_addr": "10.20.0.12:6832\/8007509",
            "heartbeat_front_addr": "10.20.0.12:6834\/8007509",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 23,
            "uuid": "e1d81949-f4b5-4cf2-b6af-dccaaeb30ed7",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17804,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6877\/7592",
            "cluster_addr": "10.20.0.12:6826\/4007592",
            "heartbeat_back_addr": "10.20.0.12:6835\/4007592",
            "heartbeat_front_addr": "10.20.0.12:6839\/4007592",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 24,
            "uuid": "ede77283-a423-4c6b-9c6e-b0e807c63cb5",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15688,
            "last_clean_end": 16109,
            "up_from": 16112,
            "up_thru": 17802,
            "down_at": 16109,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6921\/213483",
            "cluster_addr": "10.20.0.13:6815\/3213483",
            "heartbeat_back_addr": "10.20.0.13:6841\/3213483",
            "heartbeat_front_addr": "10.20.0.13:6846\/3213483",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 25,
            "uuid": "7cfe85f8-3ae9-493d-9801-025ff6c6265d",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15686,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17801,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6801\/212863",
            "cluster_addr": "10.20.0.13:6918\/3212863",
            "heartbeat_back_addr": "10.20.0.13:6932\/3212863",
            "heartbeat_front_addr": "10.20.0.13:6934\/3212863",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 26,
            "uuid": "266f6d70-519f-4c24-bca2-236495a600a7",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15692,
            "last_clean_end": 16125,
            "up_from": 16130,
            "up_thru": 17802,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6917\/213469",
            "cluster_addr": "10.20.0.13:6896\/4213469",
            "heartbeat_back_addr": "10.20.0.13:6945\/4213469",
            "heartbeat_front_addr": "10.20.0.13:6946\/4213469",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 27,
            "uuid": "68644fa9-9459-4db0-a6c9-01661645038b",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15689,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17802,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6899\/213448",
            "cluster_addr": "10.20.0.13:6876\/3213448",
            "heartbeat_back_addr": "10.20.0.13:6902\/3213448",
            "heartbeat_front_addr": "10.20.0.13:6904\/3213448",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 28,
            "uuid": "fc3d5749-7673-4100-a0d4-f25e9cc0bc88",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15688,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17801,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6892\/213419",
            "cluster_addr": "10.20.0.13:6866\/5213419",
            "heartbeat_back_addr": "10.20.0.13:6867\/5213419",
            "heartbeat_front_addr": "10.20.0.13:6873\/5213419",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 29,
            "uuid": "cb5feda9-de3f-4e42-bb73-7945b4928b22",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15689,
            "last_clean_end": 16109,
            "up_from": 16113,
            "up_thru": 17802,
            "down_at": 16109,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6937\/213527",
            "cluster_addr": "10.20.0.13:6898\/2213527",
            "heartbeat_back_addr": "10.20.0.13:6926\/2213527",
            "heartbeat_front_addr": "10.20.0.13:6927\/2213527",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 30,
            "uuid": "ef1e65bb-a634-4096-9466-1262af55db01",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15693,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17802,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6910\/213454",
            "cluster_addr": "10.20.0.13:6823\/4213454",
            "heartbeat_back_addr": "10.20.0.13:6838\/4213454",
            "heartbeat_front_addr": "10.20.0.13:6852\/4213454",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 31,
            "uuid": "3dad6393-67a8-43d4-ba8d-ffd320827396",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15688,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17802,
            "down_at": 16125,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6929\/213515",
            "cluster_addr": "10.20.0.13:6909\/4213515",
            "heartbeat_back_addr": "10.20.0.13:6940\/4213515",
            "heartbeat_front_addr": "10.20.0.13:6941\/4213515",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 32,
            "uuid": "db6f3afa-53ed-453a-97e3-861e88cb818f",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15684,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17817,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6884\/213261",
            "cluster_addr": "10.20.0.13:6856\/3213261",
            "heartbeat_back_addr": "10.20.0.13:6863\/3213261",
            "heartbeat_front_addr": "10.20.0.13:6865\/3213261",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 33,
            "uuid": "d5e59852-06b4-4a30-8c5e-ff7e328b5455",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15692,
            "last_clean_end": 16125,
            "up_from": 16130,
            "up_thru": 17803,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6900\/213450",
            "cluster_addr": "10.20.0.13:6936\/3213450",
            "heartbeat_back_addr": "10.20.0.13:6938\/3213450",
            "heartbeat_front_addr": "10.20.0.13:6939\/3213450",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 34,
            "uuid": "f35a10c5-217a-4cfb-88b9-7334bda441b8",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15687,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17802,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6831\/212981",
            "cluster_addr": "10.20.0.13:6942\/2212981",
            "heartbeat_back_addr": "10.20.0.13:6943\/2212981",
            "heartbeat_front_addr": "10.20.0.13:6944\/2212981",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 35,
            "uuid": "335e797f-a390-4f08-9da6-9ab76ffb12ae",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15687,
            "last_clean_end": 16109,
            "up_from": 16112,
            "up_thru": 17802,
            "down_at": 16109,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6877\/213258",
            "cluster_addr": "10.20.0.13:6803\/2213258",
            "heartbeat_back_addr": "10.20.0.13:6832\/2213258",
            "heartbeat_front_addr": "10.20.0.13:6833\/2213258",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 36,
            "uuid": "10be14b2-1bc9-4aaf-8d5c-1f7aed2eadc5",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15727,
            "last_clean_end": 16126,
            "up_from": 17436,
            "up_thru": 17652,
            "down_at": 17677,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6818\/448672",
            "cluster_addr": "10.20.0.11:6819\/448672",
            "heartbeat_back_addr": "10.20.0.11:6820\/448672",
            "heartbeat_front_addr": "10.20.0.11:6824\/448672",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 37,
            "uuid": "266fe6ea-0152-4584-a8e0-3fb5d0c2f04c",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15721,
            "last_clean_end": 16127,
            "up_from": 16512,
            "up_thru": 16968,
            "down_at": 17678,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6809\/440793",
            "cluster_addr": "10.20.0.11:6810\/440793",
            "heartbeat_back_addr": "10.20.0.11:6816\/440793",
            "heartbeat_front_addr": "10.20.0.11:6817\/440793",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 38,
            "uuid": "9011180a-2de1-4c3a-9f7c-e917878be18b",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15724,
            "last_clean_end": 16127,
            "up_from": 17663,
            "up_thru": 17671,
            "down_at": 17679,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6825\/451446",
            "cluster_addr": "10.20.0.11:6826\/451446",
            "heartbeat_back_addr": "10.20.0.11:6827\/451446",
            "heartbeat_front_addr": "10.20.0.11:6828\/451446",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 39,
            "uuid": "fe3819a6-ed64-4a4f-9b76-791f43d32b10",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 17689,
            "last_clean_end": 17710,
            "up_from": 17821,
            "up_thru": 17700,
            "down_at": 17835,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6820\/460064",
            "cluster_addr": "10.20.0.11:6824\/460064",
            "heartbeat_back_addr": "10.20.0.11:6825\/460064",
            "heartbeat_front_addr": "10.20.0.11:6827\/460064",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 40,
            "uuid": "5f8a9ce2-3efb-4d9c-8c2f-e45ebe6e4990",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15730,
            "last_clean_end": 16130,
            "up_from": 17687,
            "up_thru": 16130,
            "down_at": 17712,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6818\/452335",
            "cluster_addr": "10.20.0.11:6819\/452335",
            "heartbeat_back_addr": "10.20.0.11:6820\/452335",
            "heartbeat_front_addr": "10.20.0.11:6824\/452335",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 41,
            "uuid": "79791990-10e4-4437-ba86-bdf1165700c8",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15727,
            "last_clean_end": 16131,
            "up_from": 17689,
            "up_thru": 17701,
            "down_at": 17713,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6825\/452386",
            "cluster_addr": "10.20.0.11:6826\/452386",
            "heartbeat_back_addr": "10.20.0.11:6827\/452386",
            "heartbeat_front_addr": "10.20.0.11:6828\/452386",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 42,
            "uuid": "15c862bb-fc1b-41b4-8f62-15772ac8c51a",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15732,
            "last_clean_end": 16133,
            "up_from": 17725,
            "up_thru": 17738,
            "down_at": 17754,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6809\/454290",
            "cluster_addr": "10.20.0.11:6810\/454290",
            "heartbeat_back_addr": "10.20.0.11:6816\/454290",
            "heartbeat_front_addr": "10.20.0.11:6817\/454290",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 43,
            "uuid": "ae8550c7-cd08-4d32-91ea-fe1e2909ec9e",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15727,
            "last_clean_end": 16135,
            "up_from": 17722,
            "up_thru": 17732,
            "down_at": 17755,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6818\/454306",
            "cluster_addr": "10.20.0.11:6819\/454306",
            "heartbeat_back_addr": "10.20.0.11:6820\/454306",
            "heartbeat_front_addr": "10.20.0.11:6824\/454306",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 44,
            "uuid": "a10860b0-6430-4f0f-ad10-27b87b29c202",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15733,
            "last_clean_end": 16141,
            "up_from": 17724,
            "up_thru": 17743,
            "down_at": 17756,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6825\/454350",
            "cluster_addr": "10.20.0.11:6826\/454350",
            "heartbeat_back_addr": "10.20.0.11:6827\/454350",
            "heartbeat_front_addr": "10.20.0.11:6828\/454350",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 45,
            "uuid": "2046b313-4aa8-4172-8e58-46cb2d00d049",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 17766,
            "last_clean_end": 17805,
            "up_from": 17814,
            "up_thru": 17780,
            "down_at": 17837,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6809\/459655",
            "cluster_addr": "10.20.0.11:6816\/459655",
            "heartbeat_back_addr": "10.20.0.11:6817\/459655",
            "heartbeat_front_addr": "10.20.0.11:6818\/459655",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 46,
            "uuid": "9339227e-6423-43ea-a137-5221e243ff96",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15730,
            "last_clean_end": 16150,
            "up_from": 17761,
            "up_thru": 17779,
            "down_at": 17807,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6820\/456529",
            "cluster_addr": "10.20.0.11:6824\/456529",
            "heartbeat_back_addr": "10.20.0.11:6825\/456529",
            "heartbeat_front_addr": "10.20.0.11:6827\/456529",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 47,
            "uuid": "ee4aa4f0-835e-4179-bcc9-a3b8029bda07",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 17765,
            "last_clean_end": 17807,
            "up_from": 17824,
            "up_thru": 17774,
            "down_at": 17838,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6828\/460486",
            "cluster_addr": "10.20.0.11:6829\/460486",
            "heartbeat_back_addr": "10.20.0.11:6830\/460486",
            "heartbeat_front_addr": "10.20.0.11:6832\/460486",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 48,
            "uuid": "80d1c85b-f6ad-4a53-8ff6-cb15e2b2c886",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 16556,
            "last_clean_end": 17679,
            "up_from": 17815,
            "up_thru": 16556,
            "down_at": 17833,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6807\/182332",
            "cluster_addr": "10.20.0.12:6808\/182332",
            "heartbeat_back_addr": "10.20.0.12:6811\/182332",
            "heartbeat_front_addr": "10.20.0.12:6812\/182332",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 49,
            "uuid": "3f5c4970-e8d5-446e-bedb-3c4364e06de8",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16129,
            "up_from": 17502,
            "up_thru": 16068,
            "down_at": 17681,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6836\/175317",
            "cluster_addr": "10.20.0.12:6837\/175317",
            "heartbeat_back_addr": "10.20.0.12:6838\/175317",
            "heartbeat_front_addr": "10.20.0.12:6842\/175317",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 50,
            "uuid": "43c6f777-42b9-418f-bbaf-afce89a1da33",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16130,
            "up_from": 17675,
            "up_thru": 16118,
            "down_at": 17682,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6846\/176153",
            "cluster_addr": "10.20.0.12:6847\/176153",
            "heartbeat_back_addr": "10.20.0.12:6848\/176153",
            "heartbeat_front_addr": "10.20.0.12:6852\/176153",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 51,
            "uuid": "04c9dc0c-836f-4828-8386-5390dc6312c9",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16131,
            "up_from": 17695,
            "up_thru": 16126,
            "down_at": 17714,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6807\/176856",
            "cluster_addr": "10.20.0.12:6808\/176856",
            "heartbeat_back_addr": "10.20.0.12:6811\/176856",
            "heartbeat_front_addr": "10.20.0.12:6812\/176856",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 52,
            "uuid": "035e9a74-d6ce-45d5-ba3c-1178f569f27b",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16134,
            "up_from": 17695,
            "up_thru": 16072,
            "down_at": 17715,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6836\/176879",
            "cluster_addr": "10.20.0.12:6837\/176879",
            "heartbeat_back_addr": "10.20.0.12:6838\/176879",
            "heartbeat_front_addr": "10.20.0.12:6842\/176879",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 53,
            "uuid": "808fb005-0ffd-4aba-b6ca-e3c8708af1f5",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 17695,
            "last_clean_end": 17716,
            "up_from": 17822,
            "up_thru": 16117,
            "down_at": 17834,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6846\/182870",
            "cluster_addr": "10.20.0.12:6847\/182870",
            "heartbeat_back_addr": "10.20.0.12:6848\/182870",
            "heartbeat_front_addr": "10.20.0.12:6852\/182870",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 54,
            "uuid": "b1d374ff-5d1b-4690-915f-8e6f1849ea79",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16139,
            "up_from": 17731,
            "up_thru": 17731,
            "down_at": 17751,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6807\/178622",
            "cluster_addr": "10.20.0.12:6808\/178622",
            "heartbeat_back_addr": "10.20.0.12:6811\/178622",
            "heartbeat_front_addr": "10.20.0.12:6812\/178622",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 55,
            "uuid": "f6c90ac8-a30a-4a37-8c85-5270fa8b423e",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16144,
            "up_from": 17737,
            "up_thru": 16129,
            "down_at": 17753,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6836\/178648",
            "cluster_addr": "10.20.0.12:6838\/178648",
            "heartbeat_back_addr": "10.20.0.12:6842\/178648",
            "heartbeat_front_addr": "10.20.0.12:6846\/178648",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 56,
            "uuid": "6d5a3648-bb8d-4aa7-9686-16885790c5f8",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 17742,
            "last_clean_end": 17756,
            "up_from": 17820,
            "up_thru": 16116,
            "down_at": 17836,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6836\/182573",
            "cluster_addr": "10.20.0.12:6837\/182573",
            "heartbeat_back_addr": "10.20.0.12:6838\/182573",
            "heartbeat_front_addr": "10.20.0.12:6842\/182573",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 57,
            "uuid": "63338d14-910e-4d77-86c6-bdfde643736b",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16154,
            "up_from": 17769,
            "up_thru": 16139,
            "down_at": 17808,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6807\/180329",
            "cluster_addr": "10.20.0.12:6811\/180329",
            "heartbeat_back_addr": "10.20.0.12:6812\/180329",
            "heartbeat_front_addr": "10.20.0.12:6836\/180329",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 58,
            "uuid": "13199a09-0db2-4ac5-9d58-3d633b93d14f",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16159,
            "up_from": 17772,
            "up_thru": 17777,
            "down_at": 17810,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6837\/180352",
            "cluster_addr": "10.20.0.12:6842\/180352",
            "heartbeat_back_addr": "10.20.0.12:6846\/180352",
            "heartbeat_front_addr": "10.20.0.12:6847\/180352",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 59,
            "uuid": "15723089-c846-4d6d-bf23-ee9ea418ba21",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 16566,
            "last_clean_end": 17682,
            "up_from": 17826,
            "up_thru": 17353,
            "down_at": 17831,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6807\/416922",
            "cluster_addr": "10.20.0.13:6808\/416922",
            "heartbeat_back_addr": "10.20.0.13:6809\/416922",
            "heartbeat_front_addr": "10.20.0.13:6811\/416922",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 60,
            "uuid": "7d768064-4ab0-4962-a95e-1257ffa839eb",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15683,
            "last_clean_end": 16131,
            "up_from": 17662,
            "up_thru": 16066,
            "down_at": 17685,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6807\/406378",
            "cluster_addr": "10.20.0.13:6808\/406378",
            "heartbeat_back_addr": "10.20.0.13:6809\/406378",
            "heartbeat_front_addr": "10.20.0.13:6811\/406378",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 61,
            "uuid": "576d436b-19d2-4eb1-ae6e-0f19a52c78e6",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 17676,
            "last_clean_end": 17685,
            "up_from": 17823,
            "up_thru": 16080,
            "down_at": 17832,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6802\/416491",
            "cluster_addr": "10.20.0.13:6804\/416491",
            "heartbeat_back_addr": "10.20.0.13:6805\/416491",
            "heartbeat_front_addr": "10.20.0.13:6806\/416491",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 62,
            "uuid": "5d1a9351-a39d-43c3-8945-6bd551ecb988",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15682,
            "last_clean_end": 16133,
            "up_from": 17693,
            "up_thru": 16121,
            "down_at": 17719,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6802\/407884",
            "cluster_addr": "10.20.0.13:6804\/407884",
            "heartbeat_back_addr": "10.20.0.13:6805\/407884",
            "heartbeat_front_addr": "10.20.0.13:6806\/407884",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 63,
            "uuid": "ad840dee-a8f7-4074-9f53-729075712b62",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15679,
            "last_clean_end": 16141,
            "up_from": 17698,
            "up_thru": 16079,
            "down_at": 17720,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6807\/407910",
            "cluster_addr": "10.20.0.13:6808\/407910",
            "heartbeat_back_addr": "10.20.0.13:6809\/407910",
            "heartbeat_front_addr": "10.20.0.13:6811\/407910",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 64,
            "uuid": "a8b2d98a-3be7-4bfe-9187-e1fe77a2b179",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15679,
            "last_clean_end": 16143,
            "up_from": 17697,
            "up_thru": 16140,
            "down_at": 17721,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6812\/407963",
            "cluster_addr": "10.20.0.13:6813\/407963",
            "heartbeat_back_addr": "10.20.0.13:6814\/407963",
            "heartbeat_front_addr": "10.20.0.13:6816\/407963",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 65,
            "uuid": "eab0c4b3-bc56-4157-b60a-51f7b7ac4732",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15675,
            "last_clean_end": 16148,
            "up_from": 17725,
            "up_thru": 16125,
            "down_at": 17747,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6802\/410040",
            "cluster_addr": "10.20.0.13:6805\/410040",
            "heartbeat_back_addr": "10.20.0.13:6806\/410040",
            "heartbeat_front_addr": "10.20.0.13:6807\/410040",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 66,
            "uuid": "227736ac-0a48-4b7a-936b-1a9d241c3c99",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15675,
            "last_clean_end": 16153,
            "up_from": 17727,
            "up_thru": 16128,
            "down_at": 17748,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6809\/410063",
            "cluster_addr": "10.20.0.13:6811\/410063",
            "heartbeat_back_addr": "10.20.0.13:6812\/410063",
            "heartbeat_front_addr": "10.20.0.13:6814\/410063",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 67,
            "uuid": "a8c1b84a-ef5a-47ff-ad49-0c3316b97bf6",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15675,
            "last_clean_end": 16158,
            "up_from": 17723,
            "up_thru": 16089,
            "down_at": 17749,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6816\/410115",
            "cluster_addr": "10.20.0.13:6817\/410115",
            "heartbeat_back_addr": "10.20.0.13:6818\/410115",
            "heartbeat_front_addr": "10.20.0.13:6819\/410115",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 68,
            "uuid": "0d9a5e37-cc79-46a7-89f6-b5e489dfb23e",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15678,
            "last_clean_end": 16162,
            "up_from": 17760,
            "up_thru": 16070,
            "down_at": 17811,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6802\/412386",
            "cluster_addr": "10.20.0.13:6804\/412386",
            "heartbeat_back_addr": "10.20.0.13:6805\/412386",
            "heartbeat_front_addr": "10.20.0.13:6806\/412386",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 69,
            "uuid": "ad69308b-7562-4880-b73c-ca05a2d8fc2e",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15678,
            "last_clean_end": 16167,
            "up_from": 17759,
            "up_thru": 16072,
            "down_at": 17812,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6807\/412409",
            "cluster_addr": "10.20.0.13:6808\/412409",
            "heartbeat_back_addr": "10.20.0.13:6809\/412409",
            "heartbeat_front_addr": "10.20.0.13:6812\/412409",
            "state": [
                "exists"
            ]
        },
        {
            "osd": 70,
            "uuid": "d4b7ba97-3298-4fd0-9b4a-e461e88352de",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15674,
            "last_clean_end": 16172,
            "up_from": 17761,
            "up_thru": 17768,
            "down_at": 17813,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6813\/412461",
            "cluster_addr": "10.20.0.13:6814\/412461",
            "heartbeat_back_addr": "10.20.0.13:6816\/412461",
            "heartbeat_front_addr": "10.20.0.13:6817\/412461",
            "state": [
                "exists"
            ]
        }
    ],
    "osd_xinfo": [
        {
            "osd": 0,
            "down_stamp": "2015-04-26 02:00:19.572511",
            "laggy_probability": 0.927101,
            "laggy_interval": 46,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 1,
            "down_stamp": "2015-04-26 02:00:19.572511",
            "laggy_probability": 0.896128,
            "laggy_interval": 43,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 2,
            "down_stamp": "2015-04-26 02:00:19.572511",
            "laggy_probability": 0.882482,
            "laggy_interval": 68,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 3,
            "down_stamp": "2015-04-26 02:21:39.868625",
            "laggy_probability": 0.943535,
            "laggy_interval": 41,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 4,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.917737,
            "laggy_interval": 56,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 5,
            "down_stamp": "2015-04-26 02:02:17.842800",
            "laggy_probability": 0.882482,
            "laggy_interval": 15,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 6,
            "down_stamp": "2015-04-26 02:00:19.572511",
            "laggy_probability": 0.917737,
            "laggy_interval": 59,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 7,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.917737,
            "laggy_interval": 95,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 8,
            "down_stamp": "2015-04-26 04:36:29.410592",
            "laggy_probability": 0.642416,
            "laggy_interval": 58,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 9,
            "down_stamp": "2015-04-26 02:00:19.572511",
            "laggy_probability": 0.919336,
            "laggy_interval": 24,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 10,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.880998,
            "laggy_interval": 69,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 11,
            "down_stamp": "2015-04-26 04:36:35.768784",
            "laggy_probability": 0.604091,
            "laggy_interval": 32,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 12,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.849601,
            "laggy_interval": 59,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 13,
            "down_stamp": "2015-04-26 03:22:55.024663",
            "laggy_probability": 0.859651,
            "laggy_interval": 42,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 14,
            "down_stamp": "2015-04-26 03:47:02.193069",
            "laggy_probability": 0.948316,
            "laggy_interval": 30,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 15,
            "down_stamp": "2015-04-26 03:22:33.512773",
            "laggy_probability": 0.945507,
            "laggy_interval": 26,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 16,
            "down_stamp": "2015-04-26 03:22:20.137195",
            "laggy_probability": 0.948413,
            "laggy_interval": 19,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 17,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.773384,
            "laggy_interval": 72,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 18,
            "down_stamp": "2015-04-26 03:22:45.016578",
            "laggy_probability": 0.889015,
            "laggy_interval": 50,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 19,
            "down_stamp": "2015-04-26 03:22:33.512773",
            "laggy_probability": 0.931228,
            "laggy_interval": 29,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 20,
            "down_stamp": "2015-04-26 03:22:55.024663",
            "laggy_probability": 0.894947,
            "laggy_interval": 43,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 21,
            "down_stamp": "2015-04-26 03:22:41.016805",
            "laggy_probability": 0.933753,
            "laggy_interval": 14,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 22,
            "down_stamp": "2015-04-26 03:47:12.044546",
            "laggy_probability": 0.951671,
            "laggy_interval": 18,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 23,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.849840,
            "laggy_interval": 65,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 24,
            "down_stamp": "2015-04-26 02:03:26.390577",
            "laggy_probability": 0.806616,
            "laggy_interval": 47,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 25,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.810979,
            "laggy_interval": 52,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 26,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.858681,
            "laggy_interval": 132,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 27,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.796432,
            "laggy_interval": 70,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 28,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.908335,
            "laggy_interval": 39,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 29,
            "down_stamp": "2015-04-26 02:03:26.390577",
            "laggy_probability": 0.732148,
            "laggy_interval": 53,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 30,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.866609,
            "laggy_interval": 44,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 31,
            "down_stamp": "2015-04-26 02:21:39.868625",
            "laggy_probability": 0.866052,
            "laggy_interval": 46,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 32,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.812573,
            "laggy_interval": 53,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 33,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.797734,
            "laggy_interval": 62,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 34,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.681026,
            "laggy_interval": 108,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 35,
            "down_stamp": "2015-04-26 02:03:26.390577",
            "laggy_probability": 0.703038,
            "laggy_interval": 47,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 36,
            "down_stamp": "2015-04-26 04:26:34.625477",
            "laggy_probability": 0.583865,
            "laggy_interval": 14,
            "features": 1125899906842623,
            "old_weight": 0
        },
        {
            "osd": 37,
            "down_stamp": "2015-04-26 04:26:37.902343",
            "laggy_probability": 0.585348,
            "laggy_interval": 35,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 38,
            "down_stamp": "2015-04-26 04:26:41.237620",
            "laggy_probability": 0.561149,
            "laggy_interval": 37,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 39,
            "down_stamp": "2015-04-26 04:41:21.591738",
            "laggy_probability": 0.433094,
            "laggy_interval": 29,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 40,
            "down_stamp": "2015-04-26 04:31:10.523987",
            "laggy_probability": 0.641786,
            "laggy_interval": 19,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 41,
            "down_stamp": "2015-04-26 04:31:13.837051",
            "laggy_probability": 0.581195,
            "laggy_interval": 32,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 42,
            "down_stamp": "2015-04-26 04:34:14.780969",
            "laggy_probability": 0.618319,
            "laggy_interval": 45,
            "features": 1125899906842623,
            "old_weight": 0
        },
        {
            "osd": 43,
            "down_stamp": "2015-04-26 04:34:17.897798",
            "laggy_probability": 0.618319,
            "laggy_interval": 14,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 44,
            "down_stamp": "2015-04-26 04:34:20.994697",
            "laggy_probability": 0.617709,
            "laggy_interval": 37,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 45,
            "down_stamp": "2015-04-26 04:41:24.759914",
            "laggy_probability": 0.318740,
            "laggy_interval": 45,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 46,
            "down_stamp": "2015-04-26 04:37:59.919908",
            "laggy_probability": 0.583313,
            "laggy_interval": 62,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 47,
            "down_stamp": "2015-04-26 04:41:27.882318",
            "laggy_probability": 0.368083,
            "laggy_interval": 24,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 48,
            "down_stamp": "2015-04-26 04:41:16.379505",
            "laggy_probability": 0.376217,
            "laggy_interval": 36,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 49,
            "down_stamp": "2015-04-26 04:27:03.567206",
            "laggy_probability": 0.592149,
            "laggy_interval": 25,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 50,
            "down_stamp": "2015-04-26 04:27:04.676795",
            "laggy_probability": 0.555430,
            "laggy_interval": 43,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 51,
            "down_stamp": "2015-04-26 04:31:16.034637",
            "laggy_probability": 0.496930,
            "laggy_interval": 21,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 52,
            "down_stamp": "2015-04-26 04:31:19.128934",
            "laggy_probability": 0.587214,
            "laggy_interval": 22,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 53,
            "down_stamp": "2015-04-26 04:41:19.500570",
            "laggy_probability": 0.413507,
            "laggy_interval": 16,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 54,
            "down_stamp": "2015-04-26 04:34:04.259736",
            "laggy_probability": 0.584793,
            "laggy_interval": 19,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 55,
            "down_stamp": "2015-04-26 04:34:07.571042",
            "laggy_probability": 0.555430,
            "laggy_interval": 20,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 56,
            "down_stamp": "2015-04-26 04:41:22.650530",
            "laggy_probability": 0.382869,
            "laggy_interval": 13,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 57,
            "down_stamp": "2015-04-26 04:38:03.048749",
            "laggy_probability": 0.490565,
            "laggy_interval": 32,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 58,
            "down_stamp": "2015-04-26 04:38:06.220080",
            "laggy_probability": 0.546174,
            "laggy_interval": 46,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 59,
            "down_stamp": "2015-04-26 04:41:11.190236",
            "laggy_probability": 0.306253,
            "laggy_interval": 14,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 60,
            "down_stamp": "2015-04-26 04:27:18.492423",
            "laggy_probability": 0.554090,
            "laggy_interval": 17,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 61,
            "down_stamp": "2015-04-26 04:41:14.304955",
            "laggy_probability": 0.417258,
            "laggy_interval": 12,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 62,
            "down_stamp": "2015-04-26 04:31:35.273855",
            "laggy_probability": 0.597863,
            "laggy_interval": 12,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 63,
            "down_stamp": "2015-04-26 04:31:36.357348",
            "laggy_probability": 0.545616,
            "laggy_interval": 17,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 64,
            "down_stamp": "2015-04-26 04:31:39.476062",
            "laggy_probability": 0.554090,
            "laggy_interval": 31,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 65,
            "down_stamp": "2015-04-26 04:33:52.803421",
            "laggy_probability": 0.554090,
            "laggy_interval": 32,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 66,
            "down_stamp": "2015-04-26 04:33:55.914944",
            "laggy_probability": 0.554090,
            "laggy_interval": 48,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 67,
            "down_stamp": "2015-04-26 04:33:59.049520",
            "laggy_probability": 0.597863,
            "laggy_interval": 19,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 68,
            "down_stamp": "2015-04-26 04:38:11.822916",
            "laggy_probability": 0.554090,
            "laggy_interval": 16,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 69,
            "down_stamp": "2015-04-26 04:38:14.921856",
            "laggy_probability": 0.554090,
            "laggy_interval": 22,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 70,
            "down_stamp": "2015-04-26 04:38:18.015390",
            "laggy_probability": 0.554090,
            "laggy_interval": 35,
            "features": 1125899906842623,
            "old_weight": 65536
        }
    ],
    "pg_temp": [],
    "primary_temp": [],
    "blacklist": [],
    "erasure_code_profiles": {
        "default": {
            "directory": "\/usr\/lib\/ceph\/erasure-code",
            "k": "2",
            "m": "1",
            "plugin": "jerasure",
            "technique": "reed_sol_van"
        }
    }
}
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux