Here it is
Br,
Tuomas
-----Original Message-----
From: Sage Weil [mailto:sweil@xxxxxxxxxx]
Sent: 28. huhtikuuta 2015 21:57
To: Tuomas Juntunen
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: RE: Upgrade from Giant to Hammer and after some basic
operations most of the OSD's went down
On Tue, 28 Apr 2015, Tuomas Juntunen wrote:
> Hi
>
> Here's the file.
Hrm. Can you also attach
ceph osd dump -f json-pretty 17846
> The cluster was working totally fine for months before this, no
> problems at all.
>
> 'ceph osd tree' still shows 11 osd's up, even though all of them are down.
> If I add a new one, that will get added to the list and it goes up,
> but when I stop it, the status will still show it as up.
>
> Here's a log for that osd http://beta.xaasbox.com/ceph/ceph-osd.37.log
Does your ceph.conf have any changes to osd_heartbeat_* configurables?
It is very strange that the OSDs aren't getting marked down.
sage
>
> Br,
> Tuomas
>
>
> -----Original Message-----
> From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> Sent: 28. huhtikuuta 2015 20:02
> To: Tuomas Juntunen
> Cc: ceph-users@xxxxxxxxxxxxxx
> Subject: RE: Upgrade from Giant to Hammer and after some
> basic operations most of the OSD's went down
>
> On Tue, 28 Apr 2015, Tuomas Juntunen wrote:
> > Just to add some more interesting behavior to my problem, is that
> > monitors are not updating the status of OSD's.
>
> Yeah, this is very strange. I see
>
> 2015-04-27 22:25:26.142245 7f78d793a900 10 osd.15 17882 _get_pool 4
> cached_removed_snaps [1~1,4~a,f~2,12~10,23~4,28~2,2e~2,32~c,3f~4]
>
> but pool 4 (images) has no snaps and a zeroed snap_seq in the osdmap
> dump you provided. Can you attach
>
> ceph osd dump -f json-pretty 17882
>
> ?
>
> The fact that your mon status isn't changing makes me wonder if these
> are two different ceph clusters or something similarly silly... if
> the mds daemons are down the status should show stale. If all osds
> stop you might not see the osds go down for ~10 minutes (since they
> aren't monitoring each other).
>
> sage
>
> >
> > Even when I stop all the remaining OSD's, ceph osd tree shows them as
up.
> > Also there the status of mons and mds doesn't seem to update
> > correctly in my opinion.
> >
> > Below is a copy of status when one mon and mds are stopped and all
> > of the osd's are also stopped.
> >
> > monmap e7: 3 mons at
> >
{ceph1=10.20.0.11:6789/0,ceph2=10.20.0.12:6789/0,ceph3=10.20.0.13:6789/0}
> > election epoch 48, quorum 0,1 ceph1,ceph2
> > mdsmap e1750: 1/1/1 up {0=ceph2=up:replay}, 1 up:standby
> > osdmap e18132: 37 osds: 11 up, 11 in
> >
> > Br,
> > Tuomas
> >
> >
> > -----Original Message-----
> > From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> > Sent: 27. huhtikuuta 2015 22:22
> > To: Tuomas Juntunen
> > Cc: ceph-users@xxxxxxxxxxxxxx
> > Subject: RE: Upgrade from Giant to Hammer and after
> > some basic operations most of the OSD's went down
> >
> > On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> > > Hey
> > >
> > > Got the log, you can get it from
> > > http://beta.xaasbox.com/ceph/ceph-osd.15.log
> >
> > Can you repeat this with 'debug osd = 20'? Thanks!
> >
> > sage
> >
> > >
> > > Br,
> > > Tuomas
> > >
> > >
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> > > Sent: 27. huhtikuuta 2015 20:45
> > > To: Tuomas Juntunen
> > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > Subject: Re: Upgrade from Giant to Hammer and after
> > > some basic operations most of the OSD's went down
> > >
> > > Yeah, no snaps:
> > >
> > > images:
> > > "snap_mode": "selfmanaged",
> > > "snap_seq": 0,
> > > "snap_epoch": 17882,
> > > "pool_snaps": [],
> > > "removed_snaps": "[]",
> > >
> > > img:
> > > "snap_mode": "selfmanaged",
> > > "snap_seq": 0,
> > > "snap_epoch": 0,
> > > "pool_snaps": [],
> > > "removed_snaps": "[]",
> > >
> > > ...and actually the log shows this happens on pool 2 (rbd), which
> > > has
> > >
> > > "snap_mode": "selfmanaged",
> > > "snap_seq": 0,
> > > "snap_epoch": 0,
> > > "pool_snaps": [],
> > > "removed_snaps": "[]",
> > >
> > > I'm guessin gthe offending code is
> > >
> > > pi->build_removed_snaps(newly_removed_snaps);
> > > newly_removed_snaps.subtract(cached_removed_snaps);
> > >
> > > so newly_removed_snaps should be empty, and apparently
> > > cached_removed_snaps is not? Maybe one of your older osdmaps has
> > > snap info for rbd? It doesn't make sense. :/ Maybe
> > >
> > > ceph osd dump 18127 -f json-pretty
> > >
> > > just to be certain? I've pushed a branch 'wip-hammer-snaps' that
> > > will appear at gitbuilder.ceph.com in 20-30 minutes that will
> > > output some additional debug info. It will be at
> > >
> > >
> > > http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/wip-ha
> > > mm
> > > er
> > > -sanps
> > >
> > > or similar, depending on your distro. Can you install it one on
> > > node and start and osd with logging to reproduce the crash?
> > >
> > > Thanks!
> > > sage
> > >
> > >
> > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> > >
> > > > Hi
> > > >
> > > > Here you go
> > > >
> > > > Br,
> > > > Tuomas
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Sage Weil [mailto:sweil@xxxxxxxxxx]
> > > > Sent: 27. huhtikuuta 2015 19:23
> > > > To: Tuomas Juntunen
> > > > Cc: 'Samuel Just'; ceph-users@xxxxxxxxxxxxxx
> > > > Subject: Re: Upgrade from Giant to Hammer and after
> > > > some basic operations most of the OSD's went down
> > > >
> > > > On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> > > > > Thanks for the info.
> > > > >
> > > > > For my knowledge there was no snapshots on that pool, but
> > > > > cannot verify that.
> > > >
> > > > Can you attach a 'ceph osd dump -f json-pretty'? That will shed
> > > > a bit more light on what happened (and the simplest way to fix it).
> > > >
> > > > sage
> > > >
> > > >
> > > > > Any way to make this work again? Removing the tier and other
> > > > > settings didn't fix it, I tried it the second this happened.
> > > > >
> > > > > Br,
> > > > > Tuomas
> > > > >
> > > > > -----Original Message-----
> > > > > From: Samuel Just [mailto:sjust@xxxxxxxxxx]
> > > > > Sent: 27. huhtikuuta 2015 15:50
> > > > > To: tuomas juntunen
> > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > Subject: Re: Upgrade from Giant to Hammer and
> > > > > after some basic operations most of the OSD's went down
> > > > >
> > > > > So, the base tier is what determines the snapshots for the
> > > > > cache/base pool
> > > > amalgam. You added a populated pool complete with snapshots on
> > > > top of a base tier without snapshots. Apparently, it caused an
> > > > existential crisis for the snapshot code. That's one of the
> > > > reasons why there is a --force-nonempty flag for that operation,
> > > > I think. I think the immediate answer is probably to disallow
> > > > pools with snapshots as a cache tier altogether until we think
> > > > of a good way to
> > make it work.
> > > > > -Sam
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > > > > To: "Samuel Just" <sjust@xxxxxxxxxx>
> > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > Sent: Monday, April 27, 2015 4:56:58 AM
> > > > > Subject: Re: Upgrade from Giant to Hammer and
> > > > > after some basic operations most of the OSD's went down
> > > > >
> > > > >
> > > > >
> > > > > The following:
> > > > >
> > > > > ceph osd tier add img images --force-nonempty ceph osd tier
> > > > > cache-mode images forward ceph osd tier set-overlay img images
> > > > >
> > > > > Idea was to make images as a tier to img, move data to img
> > > > > then change
> > > > clients to use the new img pool.
> > > > >
> > > > > Br,
> > > > > Tuomas
> > > > >
> > > > > > Can you explain exactly what you mean by:
> > > > > >
> > > > > > "Also I created one pool for tier to be able to move data
> > > > > > without
> > > > outage."
> > > > > >
> > > > > > -Sam
> > > > > > ----- Original Message -----
> > > > > > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > > > > > To: "Ian Colle" <icolle@xxxxxxxxxx>
> > > > > > Cc: ceph-users@xxxxxxxxxxxxxx
> > > > > > Sent: Monday, April 27, 2015 4:23:44 AM
> > > > > > Subject: Re: Upgrade from Giant to Hammer and
> > > > > > after some basic operations most of the OSD's went down
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > Any solution for this yet?
> > > > > >
> > > > > > Br,
> > > > > > Tuomas
> > > > > >
> > > > > >> It looks like you may have hit
> > > > > >> http://tracker.ceph.com/issues/7915
> > > > > >>
> > > > > >> Ian R. Colle
> > > > > >> Global Director
> > > > > >> of Software Engineering
> > > > > >> Red Hat (Inktank is now part of Red Hat!)
> > > > > >> http://www.linkedin.com/in/ircolle
> > > > > >> http://www.twitter.com/ircolle
> > > > > >> Cell: +1.303.601.7713
> > > > > >> Email: icolle@xxxxxxxxxx
> > > > > >>
> > > > > >> ----- Original Message -----
> > > > > >> From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > > > > >> To: ceph-users@xxxxxxxxxxxxxx
> > > > > >> Sent: Monday, April 27, 2015 1:56:29 PM
> > > > > >> Subject: Upgrade from Giant to Hammer and
> > > > > >> after some basic operations most of the OSD's went down
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> I upgraded Ceph from 0.87 Giant to 0.94.1 Hammer
> > > > > >>
> > > > > >> Then created new pools and deleted some old ones. Also I
> > > > > >> created one pool for tier to be able to move data without
outage.
> > > > > >>
> > > > > >> After these operations all but 10 OSD's are down and
> > > > > >> creating this kind of messages to logs, I get more than
> > > > > >> 100gb of these in a
> > > night:
> > > > > >>
> > > > > >> -19> 2015-04-27 10:17:08.808584 7fd8e748d700 5 osd.23
pg_epoch:
>
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] enter Started
> > > > > >> -18> 2015-04-27 10:17:08.808596 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] enter Start
> > > > > >> -17> 2015-04-27 10:17:08.808608 7fd8e748d700 1 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
> > > > > >> -16> 2015-04-27 10:17:08.808621 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] exit Start 0.000025 0 0.000000
> > > > > >> -15> 2015-04-27 10:17:08.808637 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0
> > > > > >> ec=1 les/c
> > > > > >> 16609/16659
> > > > > >> 16590/16590/16590) [24,3,23] r=2 lpr=17838
> > > > > >> pi=15659-16589/42
> > > > > >> crt=8480'7 lcod
> > > > > >> 0'0 inactive NOTIFY] enter Started/Stray
> > > > > >> -14> 2015-04-27 10:17:08.808796 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive
> > > > > >> NOTIFY] exit Reset 0.119467 4 0.000037
> > > > > >> -13> 2015-04-27 10:17:08.808817 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive
> > > > > >> NOTIFY] enter Started
> > > > > >> -12> 2015-04-27 10:17:08.808828 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive
> > > > > >> NOTIFY] enter Start
> > > > > >> -11> 2015-04-27 10:17:08.808838 7fd8e748d700 1 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive
> > > > > >> NOTIFY]
> > > > > >> state<Start>: transitioning to Stray
> > > > > >> -10> 2015-04-27 10:17:08.808849 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive
> > > > > >> NOTIFY] exit Start 0.000020 0 0.000000
> > > > > >> -9> 2015-04-27 10:17:08.808861 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c
> > > > > >> 17879/17879
> > > > > >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive
> > > > > >> NOTIFY] enter Started/Stray
> > > > > >> -8> 2015-04-27 10:17:08.809427 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive] exit Reset 7.511623 45 0.000165
> > > > > >> -7> 2015-04-27 10:17:08.809445 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive] enter Started
> > > > > >> -6> 2015-04-27 10:17:08.809456 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive] enter Start
> > > > > >> -5> 2015-04-27 10:17:08.809468 7fd8e748d700 1 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive]
> > > > > >> state<Start>: transitioning to Primary
> > > > > >> -4> 2015-04-27 10:17:08.809479 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive] exit Start 0.000023 0 0.000000
> > > > > >> -3> 2015-04-27 10:17:08.809492 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive] enter Started/Primary
> > > > > >> -2> 2015-04-27 10:17:08.809502 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> inactive] enter Started/Primary/Peering
> > > > > >> -1> 2015-04-27 10:17:08.809513 7fd8e748d700 5 osd.23
> pg_epoch:
> >
> > > > > >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c
> > > > > >> 16127/16344
> > > > > >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0
> > > > > >> peering] enter Started/Primary/Peering/GetInfo
> > > > > >> 0> 2015-04-27 10:17:08.813837 7fd8e748d700 -1
> > > > ./include/interval_set.h:
> > > > > >> In
> > > > > >> function 'void interval_set<T>::erase(T, T) [with T =
snapid_t]'
> > > > > >> thread
> > > > > >> 7fd8e748d700 time 2015-04-27 10:17:08.809899
> > > > > >> ./include/interval_set.h: 385: FAILED assert(_size >= 0)
> > > > > >>
> > > > > >> ceph version 0.94.1
> > > > > >> (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
> > > > > >> 1: (ceph::__ceph_assert_fail(char const*, char const*,
> > > > > >> int, char
> > > > > >> const*)+0x8b)
> > > > > >> [0xbc271b]
> > > > > >> 2:
> > > > > >> (interval_set<snapid_t>::subtract(interval_set<snapid_t>
> > > > > >> const&)+0xb0) [0x82cd50]
> > > > > >> 3: (PGPool::update(std::tr1::shared_ptr<OSDMap
> > > > > >> const>)+0x52e) [0x80113e]
> > > > > >> 4: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap
> > > > > >> const>, std::tr1::shared_ptr<OSDMap const>,
> > > > > >> const>std::vector<int,
> > > > > >> std::allocator<int> >&, int, std::vector<int,
> > > > > >> std::allocator<int>
> > > > > >> >&, int, PG::RecoveryCtx*)+0x282) [0x801652]
> > > > > >> 5: (OSD::advance_pg(unsigned int, PG*,
> > > > > >> ThreadPool::TPHandle&, PG::RecoveryCtx*,
> > > > > >> std::set<boost::intrusive_ptr<PG>,
> > > > > >> std::less<boost::intrusive_ptr<PG> >,
> > > > > >> std::allocator<boost::intrusive_ptr<PG> > >*)+0x2c3)
> > > > > >> [0x6b0e43]
> > > > > >> 6: (OSD::process_peering_events(std::list<PG*,
> > > > > >> std::allocator<PG*>
> > > > > >> > const&,
> > > > > >> ThreadPool::TPHandle&)+0x21c) [0x6b191c]
> > > > > >> 7: (OSD::PeeringWQ::_process(std::list<PG*,
> > > > > >> std::allocator<PG*>
> > > > > >> > const&,
> > > > > >> ThreadPool::TPHandle&)+0x18) [0x709278]
> > > > > >> 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e)
> > > > > >> [0xbb38ae]
> > > > > >> 9: (ThreadPool::WorkThread::entry()+0x10) [0xbb4950]
> > > > > >> 10: (()+0x8182) [0x7fd906946182]
> > > > > >> 11: (clone()+0x6d) [0x7fd904eb147d]
> > > > > >>
> > > > > >> Also by monitoring (ceph -w) I get the following messages,
> > > > > >> also lots of
> > > > them.
> > > > > >>
> > > > > >> 2015-04-27 10:39:52.935812 mon.0 [INF] from='client.?
> > > > 10.20.0.13:0/1174409'
> > > > > >> entity='osd.30' cmd=[{"prefix": "osd crush create-or-move",
> "args":
> > > > > >> ["host=ceph3", "root=default"], "id": 30, "weight": 1.82}]:
> > > > > >> dispatch
> > > > > >> 2015-04-27 10:39:53.297376 mon.0 [INF] from='client.?
> > > > 10.20.0.13:0/1174483'
> > > > > >> entity='osd.26' cmd=[{"prefix": "osd crush create-or-move",
> "args":
> > > > > >> ["host=ceph3", "root=default"], "id": 26, "weight": 1.82}]:
> > > > > >> dispatch
> > > > > >>
> > > > > >>
> > > > > >> This is a cluster of 3 nodes with 36 OSD's, nodes are also
> > > > > >> mons and mds's to save servers. All run Ubuntu 14.04.2.
> > > > > >>
> > > > > >> I have pretty much tried everything I could think of.
> > > > > >>
> > > > > >> Restarting daemons doesn't help.
> > > > > >>
> > > > > >> Any help would be appreciated. I can also provide more logs
> > > > > >> if necessary. They just seem to get pretty large in few
moments.
> > > > > >>
> > > > > >> Thank you
> > > > > >> Tuomas
> > > > > >>
> > > > > >>
> > > > > >> _______________________________________________
> > > > > >> ceph-users mailing list
> > > > > >> ceph-users@xxxxxxxxxxxxxx
> > > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list
> > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
>
{
"epoch": 17846,
"fsid": "a2974742-3805-4cd3-bc79-765f2bddaefe",
"created": "2014-10-15 20:43:45.186949",
"modified": "2015-04-26 10:39:35.366678",
"flags": "",
"cluster_snapshot": "",
"pool_max": 7,
"max_osd": 71,
"pools": [
{
"pool": 0,
"pool_name": "data",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 45,
"last_change": "1112",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 0,
"snap_epoch": 0,
"pool_snaps": [],
"removed_snaps": "[]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 0,
"cache_target_full_ratio_micro": 0,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 1,
"pool_name": "metadata",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "1114",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 0,
"snap_epoch": 0,
"pool_snaps": [],
"removed_snaps": "[]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 0,
"cache_target_full_ratio_micro": 0,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 2,
"pool_name": "rbd",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 2,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "1116",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 0,
"snap_epoch": 0,
"pool_snaps": [],
"removed_snaps": "[]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 0,
"cache_target_full_ratio_micro": 0,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 3,
"pool_name": "volumes",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "9974",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 23,
"snap_epoch": 9974,
"pool_snaps": [],
"removed_snaps": "[1~17]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 400000,
"cache_target_full_ratio_micro": 800000,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "default",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 4,
"pool_name": "images",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "17846",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 66,
"snap_epoch": 17846,
"pool_snaps": [],
"removed_snaps": "[1~1,4~a,f~2,12~10,23~4,28~2,2e~2,32~c,3f~4]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 400000,
"cache_target_full_ratio_micro": 800000,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "default",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 5,
"pool_name": "backups",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "1121",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 3,
"snap_epoch": 705,
"pool_snaps": [],
"removed_snaps": "[1~3]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 400000,
"cache_target_full_ratio_micro": 800000,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "default",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 6,
"pool_name": "vms",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "1122",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 0,
"snap_epoch": 0,
"pool_snaps": [],
"removed_snaps": "[]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 400000,
"cache_target_full_ratio_micro": 800000,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "default",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 1,
"stripe_width": 0,
"expected_num_objects": 0
},
{
"pool": 7,
"pool_name": "san",
"flags": 1,
"flags_names": "hashpspool",
"type": 1,
"size": 3,
"min_size": 1,
"crush_ruleset": 0,
"object_hash": 2,
"pg_num": 4096,
"pg_placement_num": 4096,
"crash_replay_interval": 0,
"last_change": "14096",
"last_force_op_resend": "0",
"auid": 0,
"snap_mode": "selfmanaged",
"snap_seq": 0,
"snap_epoch": 0,
"pool_snaps": [],
"removed_snaps": "[]",
"quota_max_bytes": 0,
"quota_max_objects": 0,
"tiers": [],
"tier_of": -1,
"read_tier": -1,
"write_tier": -1,
"cache_mode": "none",
"target_max_bytes": 0,
"target_max_objects": 0,
"cache_target_dirty_ratio_micro": 400000,
"cache_target_full_ratio_micro": 800000,
"cache_min_flush_age": 0,
"cache_min_evict_age": 0,
"erasure_code_profile": "",
"hit_set_params": {
"type": "none"
},
"hit_set_period": 0,
"hit_set_count": 0,
"min_read_recency_for_promote": 0,
"stripe_width": 0,
"expected_num_objects": 0
}
],
"osds": [
{
"osd": 0,
"uuid": "757c3bc5-4d00-4344-8de4-82f5379c96af",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15738,
"last_clean_end": 16079,
"up_from": 16081,
"up_thru": 17776,
"down_at": 16079,
"lost_at": 0,
"public_addr": "10.20.0.11:6981\/245797",
"cluster_addr": "10.20.0.11:6881\/5245797",
"heartbeat_back_addr": "10.20.0.11:6883\/5245797",
"heartbeat_front_addr": "10.20.0.11:6887\/5245797",
"state": [
"exists",
"up"
]
},
{
"osd": 1,
"uuid": "c7eaa4ac-99fc-46db-84aa-a67274896ec8",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15740,
"last_clean_end": 16079,
"up_from": 16083,
"up_thru": 17781,
"down_at": 16079,
"lost_at": 0,
"public_addr": "10.20.0.11:6813\/245273",
"cluster_addr": "10.20.0.11:6888\/4245273",
"heartbeat_back_addr": "10.20.0.11:6890\/4245273",
"heartbeat_front_addr": "10.20.0.11:6894\/4245273",
"state": [
"exists",
"up"
]
},
{
"osd": 2,
"uuid": "206b2949-4adf-4789-8e06-f68a8ee819c9",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15739,
"last_clean_end": 16079,
"up_from": 16081,
"up_thru": 17774,
"down_at": 16079,
"lost_at": 0,
"public_addr": "10.20.0.11:6808\/245269",
"cluster_addr": "10.20.0.11:6811\/4245269",
"heartbeat_back_addr": "10.20.0.11:6823\/4245269",
"heartbeat_front_addr": "10.20.0.11:6831\/4245269",
"state": [
"exists",
"up"
]
},
{
"osd": 3,
"uuid": "90b7c219-4dcd-48ea-a24d-f3b796a521e4",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15736,
"last_clean_end": 16125,
"up_from": 16128,
"up_thru": 17774,
"down_at": 16125,
"lost_at": 0,
"public_addr": "10.20.0.11:6848\/245357",
"cluster_addr": "10.20.0.11:6886\/6245357",
"heartbeat_back_addr": "10.20.0.11:6891\/6245357",
"heartbeat_front_addr": "10.20.0.11:6892\/6245357",
"state": [
"exists",
"up"
]
},
{
"osd": 4,
"uuid": "049ef94f-121a-4e71-8ba6-27eaebf0a569",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15737,
"last_clean_end": 16123,
"up_from": 16125,
"up_thru": 17774,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.11:6854\/245375",
"cluster_addr": "10.20.0.11:6849\/5245375",
"heartbeat_back_addr": "10.20.0.11:6850\/5245375",
"heartbeat_front_addr": "10.20.0.11:6853\/5245375",
"state": [
"exists",
"up"
]
},
{
"osd": 5,
"uuid": "2437a53b-339e-45af-b4de-0fc675d27405",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15734,
"last_clean_end": 16087,
"up_from": 16089,
"up_thru": 17771,
"down_at": 16087,
"lost_at": 0,
"public_addr": "10.20.0.11:6834\/245323",
"cluster_addr": "10.20.0.11:6803\/4245323",
"heartbeat_back_addr": "10.20.0.11:6804\/4245323",
"heartbeat_front_addr": "10.20.0.11:6805\/4245323",
"state": [
"exists",
"up"
]
},
{
"osd": 6,
"uuid": "f117ceed-b1fd-4069-99fe-b7aba9f3ef8d",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15738,
"last_clean_end": 16079,
"up_from": 16083,
"up_thru": 17778,
"down_at": 16079,
"lost_at": 0,
"public_addr": "10.20.0.11:6859\/245401",
"cluster_addr": "10.20.0.11:6865\/5245401",
"heartbeat_back_addr": "10.20.0.11:6868\/5245401",
"heartbeat_front_addr": "10.20.0.11:6874\/5245401",
"state": [
"exists",
"up"
]
},
{
"osd": 7,
"uuid": "e98e9b8a-9c62-4e3e-bdb4-c2c30103c0c1",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15730,
"last_clean_end": 16123,
"up_from": 16125,
"up_thru": 17771,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.11:6801\/245246",
"cluster_addr": "10.20.0.11:6812\/5245246",
"heartbeat_back_addr": "10.20.0.11:6814\/5245246",
"heartbeat_front_addr": "10.20.0.11:6821\/5245246",
"state": [
"exists",
"up"
]
},
{
"osd": 8,
"uuid": "41e471cd-fafe-4422-8bf5-22018bbe1375",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15736,
"last_clean_end": 17783,
"up_from": 17795,
"up_thru": 17799,
"down_at": 17784,
"lost_at": 0,
"public_addr": "10.20.0.11:6810\/458698",
"cluster_addr": "10.20.0.11:6819\/458698",
"heartbeat_back_addr": "10.20.0.11:6826\/458698",
"heartbeat_front_addr": "10.20.0.11:6833\/458698",
"state": [
"exists",
"up"
]
},
{
"osd": 9,
"uuid": "d68eeebd-d058-4b1c-a30a-994bf8fc8030",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15733,
"last_clean_end": 16079,
"up_from": 16081,
"up_thru": 17733,
"down_at": 16079,
"lost_at": 0,
"public_addr": "10.20.0.11:6969\/245600",
"cluster_addr": "10.20.0.11:6847\/5245600",
"heartbeat_back_addr": "10.20.0.11:6860\/5245600",
"heartbeat_front_addr": "10.20.0.11:6864\/5245600",
"state": [
"exists",
"up"
]
},
{
"osd": 10,
"uuid": "660747d6-3f47-449a-bc69-5399b0d54ff6",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15736,
"last_clean_end": 16124,
"up_from": 16128,
"up_thru": 17781,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.11:6822\/245281",
"cluster_addr": "10.20.0.11:6855\/4245281",
"heartbeat_back_addr": "10.20.0.11:6856\/4245281",
"heartbeat_front_addr": "10.20.0.11:6866\/4245281",
"state": [
"exists",
"up"
]
},
{
"osd": 11,
"uuid": "805965b1-127f-44a6-9a05-8a643eb7a512",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15737,
"last_clean_end": 17786,
"up_from": 17801,
"up_thru": 17804,
"down_at": 17787,
"lost_at": 0,
"public_addr": "10.20.0.11:6835\/458813",
"cluster_addr": "10.20.0.11:6836\/458813",
"heartbeat_back_addr": "10.20.0.11:6837\/458813",
"heartbeat_front_addr": "10.20.0.11:6838\/458813",
"state": [
"exists",
"up"
]
},
{
"osd": 12,
"uuid": "61fbfcbe-d642-478f-9620-f9d72ee96238",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16126,
"up_from": 16129,
"up_thru": 17818,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.12:6873\/7542",
"cluster_addr": "10.20.0.12:6947\/4007542",
"heartbeat_back_addr": "10.20.0.12:6948\/4007542",
"heartbeat_front_addr": "10.20.0.12:6949\/4007542",
"state": [
"exists",
"up"
]
},
{
"osd": 13,
"uuid": "6faad33b-00be-42a4-92ba-08be5ab7f995",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16705,
"up_from": 16707,
"up_thru": 17829,
"down_at": 16703,
"lost_at": 0,
"public_addr": "10.20.0.12:6821\/7274",
"cluster_addr": "10.20.0.12:6859\/5007274",
"heartbeat_back_addr": "10.20.0.12:6860\/5007274",
"heartbeat_front_addr": "10.20.0.12:6861\/5007274",
"state": [
"exists",
"up"
]
},
{
"osd": 14,
"uuid": "f301705b-e725-443d-96e1-d9ec9aafe657",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16962,
"up_from": 16966,
"up_thru": 17801,
"down_at": 16962,
"lost_at": 0,
"public_addr": "10.20.0.12:6809\/7209",
"cluster_addr": "10.20.0.12:6803\/7007209",
"heartbeat_back_addr": "10.20.0.12:6804\/7007209",
"heartbeat_front_addr": "10.20.0.12:6810\/7007209",
"state": [
"exists",
"up"
]
},
{
"osd": 15,
"uuid": "536bf483-10de-44b0-8e1e-4f349fbe572a",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15785,
"last_clean_end": 16700,
"up_from": 16715,
"up_thru": 17801,
"down_at": 16693,
"lost_at": 0,
"public_addr": "10.20.0.12:6881\/7613",
"cluster_addr": "10.20.0.12:6845\/7007613",
"heartbeat_back_addr": "10.20.0.12:6849\/7007613",
"heartbeat_front_addr": "10.20.0.12:6850\/7007613",
"state": [
"exists",
"up"
]
},
{
"osd": 16,
"uuid": "4185bd20-8eb0-4616-b36e-bacb181ae40e",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16688,
"up_from": 16693,
"up_thru": 17801,
"down_at": 16686,
"lost_at": 0,
"public_addr": "10.20.0.12:6813\/7210",
"cluster_addr": "10.20.0.12:6814\/7007210",
"heartbeat_back_addr": "10.20.0.12:6815\/7007210",
"heartbeat_front_addr": "10.20.0.12:6816\/7007210",
"state": [
"exists",
"up"
]
},
{
"osd": 17,
"uuid": "a6f2f5b4-477f-48f9-9acf-d5b7a6c88b98",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16123,
"up_from": 16125,
"up_thru": 17803,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.12:6817\/7215",
"cluster_addr": "10.20.0.12:6840\/3007215",
"heartbeat_back_addr": "10.20.0.12:6843\/3007215",
"heartbeat_front_addr": "10.20.0.12:6844\/3007215",
"state": [
"exists",
"up"
]
},
{
"osd": 18,
"uuid": "b31b0bd8-938a-496d-91bc-19bf4f794f82",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16700,
"up_from": 16712,
"up_thru": 17801,
"down_at": 16700,
"lost_at": 0,
"public_addr": "10.20.0.12:6805\/7185",
"cluster_addr": "10.20.0.12:6829\/5007185",
"heartbeat_back_addr": "10.20.0.12:6830\/5007185",
"heartbeat_front_addr": "10.20.0.12:6841\/5007185",
"state": [
"exists",
"up"
]
},
{
"osd": 19,
"uuid": "d76b6bd5-1ef3-436c-a75d-3587c515eb56",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16693,
"up_from": 16706,
"up_thru": 17801,
"down_at": 16693,
"lost_at": 0,
"public_addr": "10.20.0.12:6885\/7622",
"cluster_addr": "10.20.0.12:6802\/7007622",
"heartbeat_back_addr": "10.20.0.12:6818\/7007622",
"heartbeat_front_addr": "10.20.0.12:6819\/7007622",
"state": [
"exists",
"up"
]
},
{
"osd": 20,
"uuid": "8e4dd982-a4c5-4ca5-9fc5-243f55c4db57",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16703,
"up_from": 16705,
"up_thru": 17801,
"down_at": 16703,
"lost_at": 0,
"public_addr": "10.20.0.12:6825\/7316",
"cluster_addr": "10.20.0.12:6827\/5007316",
"heartbeat_back_addr": "10.20.0.12:6831\/5007316",
"heartbeat_front_addr": "10.20.0.12:6851\/5007316",
"state": [
"exists",
"up"
]
},
{
"osd": 21,
"uuid": "760aaf28-0a34-4bbc-af0c-2654b0a43fff",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16699,
"up_from": 16701,
"up_thru": 17801,
"down_at": 16698,
"lost_at": 0,
"public_addr": "10.20.0.12:6801\/7180",
"cluster_addr": "10.20.0.12:6820\/7007180",
"heartbeat_back_addr": "10.20.0.12:6822\/7007180",
"heartbeat_front_addr": "10.20.0.12:6828\/7007180",
"state": [
"exists",
"up"
]
},
{
"osd": 22,
"uuid": "40322a34-ab31-4760-b71e-a7672f812cb3",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16966,
"up_from": 16969,
"up_thru": 17801,
"down_at": 16964,
"lost_at": 0,
"public_addr": "10.20.0.12:6869\/7509",
"cluster_addr": "10.20.0.12:6824\/8007509",
"heartbeat_back_addr": "10.20.0.12:6832\/8007509",
"heartbeat_front_addr": "10.20.0.12:6834\/8007509",
"state": [
"exists",
"up"
]
},
{
"osd": 23,
"uuid": "e1d81949-f4b5-4cf2-b6af-dccaaeb30ed7",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16123,
"up_from": 16125,
"up_thru": 17804,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.12:6877\/7592",
"cluster_addr": "10.20.0.12:6826\/4007592",
"heartbeat_back_addr": "10.20.0.12:6835\/4007592",
"heartbeat_front_addr": "10.20.0.12:6839\/4007592",
"state": [
"exists",
"up"
]
},
{
"osd": 24,
"uuid": "ede77283-a423-4c6b-9c6e-b0e807c63cb5",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15688,
"last_clean_end": 16109,
"up_from": 16112,
"up_thru": 17802,
"down_at": 16109,
"lost_at": 0,
"public_addr": "10.20.0.13:6921\/213483",
"cluster_addr": "10.20.0.13:6815\/3213483",
"heartbeat_back_addr": "10.20.0.13:6841\/3213483",
"heartbeat_front_addr": "10.20.0.13:6846\/3213483",
"state": [
"exists",
"up"
]
},
{
"osd": 25,
"uuid": "7cfe85f8-3ae9-493d-9801-025ff6c6265d",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15686,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17801,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6801\/212863",
"cluster_addr": "10.20.0.13:6918\/3212863",
"heartbeat_back_addr": "10.20.0.13:6932\/3212863",
"heartbeat_front_addr": "10.20.0.13:6934\/3212863",
"state": [
"exists",
"up"
]
},
{
"osd": 26,
"uuid": "266f6d70-519f-4c24-bca2-236495a600a7",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15692,
"last_clean_end": 16125,
"up_from": 16130,
"up_thru": 17802,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6917\/213469",
"cluster_addr": "10.20.0.13:6896\/4213469",
"heartbeat_back_addr": "10.20.0.13:6945\/4213469",
"heartbeat_front_addr": "10.20.0.13:6946\/4213469",
"state": [
"exists",
"up"
]
},
{
"osd": 27,
"uuid": "68644fa9-9459-4db0-a6c9-01661645038b",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15689,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17802,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6899\/213448",
"cluster_addr": "10.20.0.13:6876\/3213448",
"heartbeat_back_addr": "10.20.0.13:6902\/3213448",
"heartbeat_front_addr": "10.20.0.13:6904\/3213448",
"state": [
"exists",
"up"
]
},
{
"osd": 28,
"uuid": "fc3d5749-7673-4100-a0d4-f25e9cc0bc88",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15688,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17801,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6892\/213419",
"cluster_addr": "10.20.0.13:6866\/5213419",
"heartbeat_back_addr": "10.20.0.13:6867\/5213419",
"heartbeat_front_addr": "10.20.0.13:6873\/5213419",
"state": [
"exists",
"up"
]
},
{
"osd": 29,
"uuid": "cb5feda9-de3f-4e42-bb73-7945b4928b22",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15689,
"last_clean_end": 16109,
"up_from": 16113,
"up_thru": 17802,
"down_at": 16109,
"lost_at": 0,
"public_addr": "10.20.0.13:6937\/213527",
"cluster_addr": "10.20.0.13:6898\/2213527",
"heartbeat_back_addr": "10.20.0.13:6926\/2213527",
"heartbeat_front_addr": "10.20.0.13:6927\/2213527",
"state": [
"exists",
"up"
]
},
{
"osd": 30,
"uuid": "ef1e65bb-a634-4096-9466-1262af55db01",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15693,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17802,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6910\/213454",
"cluster_addr": "10.20.0.13:6823\/4213454",
"heartbeat_back_addr": "10.20.0.13:6838\/4213454",
"heartbeat_front_addr": "10.20.0.13:6852\/4213454",
"state": [
"exists",
"up"
]
},
{
"osd": 31,
"uuid": "3dad6393-67a8-43d4-ba8d-ffd320827396",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15688,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17802,
"down_at": 16125,
"lost_at": 0,
"public_addr": "10.20.0.13:6929\/213515",
"cluster_addr": "10.20.0.13:6909\/4213515",
"heartbeat_back_addr": "10.20.0.13:6940\/4213515",
"heartbeat_front_addr": "10.20.0.13:6941\/4213515",
"state": [
"exists",
"up"
]
},
{
"osd": 32,
"uuid": "db6f3afa-53ed-453a-97e3-861e88cb818f",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15684,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17817,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6884\/213261",
"cluster_addr": "10.20.0.13:6856\/3213261",
"heartbeat_back_addr": "10.20.0.13:6863\/3213261",
"heartbeat_front_addr": "10.20.0.13:6865\/3213261",
"state": [
"exists",
"up"
]
},
{
"osd": 33,
"uuid": "d5e59852-06b4-4a30-8c5e-ff7e328b5455",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15692,
"last_clean_end": 16125,
"up_from": 16130,
"up_thru": 17803,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6900\/213450",
"cluster_addr": "10.20.0.13:6936\/3213450",
"heartbeat_back_addr": "10.20.0.13:6938\/3213450",
"heartbeat_front_addr": "10.20.0.13:6939\/3213450",
"state": [
"exists",
"up"
]
},
{
"osd": 34,
"uuid": "f35a10c5-217a-4cfb-88b9-7334bda441b8",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15687,
"last_clean_end": 16125,
"up_from": 16127,
"up_thru": 17802,
"down_at": 16123,
"lost_at": 0,
"public_addr": "10.20.0.13:6831\/212981",
"cluster_addr": "10.20.0.13:6942\/2212981",
"heartbeat_back_addr": "10.20.0.13:6943\/2212981",
"heartbeat_front_addr": "10.20.0.13:6944\/2212981",
"state": [
"exists",
"up"
]
},
{
"osd": 35,
"uuid": "335e797f-a390-4f08-9da6-9ab76ffb12ae",
"up": 1,
"in": 1,
"weight": 1.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15687,
"last_clean_end": 16109,
"up_from": 16112,
"up_thru": 17802,
"down_at": 16109,
"lost_at": 0,
"public_addr": "10.20.0.13:6877\/213258",
"cluster_addr": "10.20.0.13:6803\/2213258",
"heartbeat_back_addr": "10.20.0.13:6832\/2213258",
"heartbeat_front_addr": "10.20.0.13:6833\/2213258",
"state": [
"exists",
"up"
]
},
{
"osd": 36,
"uuid": "10be14b2-1bc9-4aaf-8d5c-1f7aed2eadc5",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15727,
"last_clean_end": 16126,
"up_from": 17436,
"up_thru": 17652,
"down_at": 17677,
"lost_at": 0,
"public_addr": "10.20.0.11:6818\/448672",
"cluster_addr": "10.20.0.11:6819\/448672",
"heartbeat_back_addr": "10.20.0.11:6820\/448672",
"heartbeat_front_addr": "10.20.0.11:6824\/448672",
"state": [
"exists"
]
},
{
"osd": 37,
"uuid": "266fe6ea-0152-4584-a8e0-3fb5d0c2f04c",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15721,
"last_clean_end": 16127,
"up_from": 16512,
"up_thru": 16968,
"down_at": 17678,
"lost_at": 0,
"public_addr": "10.20.0.11:6809\/440793",
"cluster_addr": "10.20.0.11:6810\/440793",
"heartbeat_back_addr": "10.20.0.11:6816\/440793",
"heartbeat_front_addr": "10.20.0.11:6817\/440793",
"state": [
"exists"
]
},
{
"osd": 38,
"uuid": "9011180a-2de1-4c3a-9f7c-e917878be18b",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15724,
"last_clean_end": 16127,
"up_from": 17663,
"up_thru": 17671,
"down_at": 17679,
"lost_at": 0,
"public_addr": "10.20.0.11:6825\/451446",
"cluster_addr": "10.20.0.11:6826\/451446",
"heartbeat_back_addr": "10.20.0.11:6827\/451446",
"heartbeat_front_addr": "10.20.0.11:6828\/451446",
"state": [
"exists"
]
},
{
"osd": 39,
"uuid": "fe3819a6-ed64-4a4f-9b76-791f43d32b10",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 17689,
"last_clean_end": 17710,
"up_from": 17821,
"up_thru": 17700,
"down_at": 17835,
"lost_at": 0,
"public_addr": "10.20.0.11:6820\/460064",
"cluster_addr": "10.20.0.11:6824\/460064",
"heartbeat_back_addr": "10.20.0.11:6825\/460064",
"heartbeat_front_addr": "10.20.0.11:6827\/460064",
"state": [
"exists"
]
},
{
"osd": 40,
"uuid": "5f8a9ce2-3efb-4d9c-8c2f-e45ebe6e4990",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15730,
"last_clean_end": 16130,
"up_from": 17687,
"up_thru": 16130,
"down_at": 17712,
"lost_at": 0,
"public_addr": "10.20.0.11:6818\/452335",
"cluster_addr": "10.20.0.11:6819\/452335",
"heartbeat_back_addr": "10.20.0.11:6820\/452335",
"heartbeat_front_addr": "10.20.0.11:6824\/452335",
"state": [
"exists"
]
},
{
"osd": 41,
"uuid": "79791990-10e4-4437-ba86-bdf1165700c8",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15727,
"last_clean_end": 16131,
"up_from": 17689,
"up_thru": 17701,
"down_at": 17713,
"lost_at": 0,
"public_addr": "10.20.0.11:6825\/452386",
"cluster_addr": "10.20.0.11:6826\/452386",
"heartbeat_back_addr": "10.20.0.11:6827\/452386",
"heartbeat_front_addr": "10.20.0.11:6828\/452386",
"state": [
"exists"
]
},
{
"osd": 42,
"uuid": "15c862bb-fc1b-41b4-8f62-15772ac8c51a",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15732,
"last_clean_end": 16133,
"up_from": 17725,
"up_thru": 17738,
"down_at": 17754,
"lost_at": 0,
"public_addr": "10.20.0.11:6809\/454290",
"cluster_addr": "10.20.0.11:6810\/454290",
"heartbeat_back_addr": "10.20.0.11:6816\/454290",
"heartbeat_front_addr": "10.20.0.11:6817\/454290",
"state": [
"exists"
]
},
{
"osd": 43,
"uuid": "ae8550c7-cd08-4d32-91ea-fe1e2909ec9e",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15727,
"last_clean_end": 16135,
"up_from": 17722,
"up_thru": 17732,
"down_at": 17755,
"lost_at": 0,
"public_addr": "10.20.0.11:6818\/454306",
"cluster_addr": "10.20.0.11:6819\/454306",
"heartbeat_back_addr": "10.20.0.11:6820\/454306",
"heartbeat_front_addr": "10.20.0.11:6824\/454306",
"state": [
"exists"
]
},
{
"osd": 44,
"uuid": "a10860b0-6430-4f0f-ad10-27b87b29c202",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15733,
"last_clean_end": 16141,
"up_from": 17724,
"up_thru": 17743,
"down_at": 17756,
"lost_at": 0,
"public_addr": "10.20.0.11:6825\/454350",
"cluster_addr": "10.20.0.11:6826\/454350",
"heartbeat_back_addr": "10.20.0.11:6827\/454350",
"heartbeat_front_addr": "10.20.0.11:6828\/454350",
"state": [
"exists"
]
},
{
"osd": 45,
"uuid": "2046b313-4aa8-4172-8e58-46cb2d00d049",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 17766,
"last_clean_end": 17805,
"up_from": 17814,
"up_thru": 17780,
"down_at": 17837,
"lost_at": 0,
"public_addr": "10.20.0.11:6809\/459655",
"cluster_addr": "10.20.0.11:6816\/459655",
"heartbeat_back_addr": "10.20.0.11:6817\/459655",
"heartbeat_front_addr": "10.20.0.11:6818\/459655",
"state": [
"exists"
]
},
{
"osd": 46,
"uuid": "9339227e-6423-43ea-a137-5221e243ff96",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15730,
"last_clean_end": 16150,
"up_from": 17761,
"up_thru": 17779,
"down_at": 17807,
"lost_at": 0,
"public_addr": "10.20.0.11:6820\/456529",
"cluster_addr": "10.20.0.11:6824\/456529",
"heartbeat_back_addr": "10.20.0.11:6825\/456529",
"heartbeat_front_addr": "10.20.0.11:6827\/456529",
"state": [
"exists"
]
},
{
"osd": 47,
"uuid": "ee4aa4f0-835e-4179-bcc9-a3b8029bda07",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 17765,
"last_clean_end": 17807,
"up_from": 17824,
"up_thru": 17774,
"down_at": 17838,
"lost_at": 0,
"public_addr": "10.20.0.11:6828\/460486",
"cluster_addr": "10.20.0.11:6829\/460486",
"heartbeat_back_addr": "10.20.0.11:6830\/460486",
"heartbeat_front_addr": "10.20.0.11:6832\/460486",
"state": [
"exists"
]
},
{
"osd": 48,
"uuid": "80d1c85b-f6ad-4a53-8ff6-cb15e2b2c886",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 16556,
"last_clean_end": 17679,
"up_from": 17815,
"up_thru": 16556,
"down_at": 17833,
"lost_at": 0,
"public_addr": "10.20.0.12:6807\/182332",
"cluster_addr": "10.20.0.12:6808\/182332",
"heartbeat_back_addr": "10.20.0.12:6811\/182332",
"heartbeat_front_addr": "10.20.0.12:6812\/182332",
"state": [
"exists"
]
},
{
"osd": 49,
"uuid": "3f5c4970-e8d5-446e-bedb-3c4364e06de8",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16129,
"up_from": 17502,
"up_thru": 16068,
"down_at": 17681,
"lost_at": 0,
"public_addr": "10.20.0.12:6836\/175317",
"cluster_addr": "10.20.0.12:6837\/175317",
"heartbeat_back_addr": "10.20.0.12:6838\/175317",
"heartbeat_front_addr": "10.20.0.12:6842\/175317",
"state": [
"exists"
]
},
{
"osd": 50,
"uuid": "43c6f777-42b9-418f-bbaf-afce89a1da33",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16130,
"up_from": 17675,
"up_thru": 16118,
"down_at": 17682,
"lost_at": 0,
"public_addr": "10.20.0.12:6846\/176153",
"cluster_addr": "10.20.0.12:6847\/176153",
"heartbeat_back_addr": "10.20.0.12:6848\/176153",
"heartbeat_front_addr": "10.20.0.12:6852\/176153",
"state": [
"exists"
]
},
{
"osd": 51,
"uuid": "04c9dc0c-836f-4828-8386-5390dc6312c9",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16131,
"up_from": 17695,
"up_thru": 16126,
"down_at": 17714,
"lost_at": 0,
"public_addr": "10.20.0.12:6807\/176856",
"cluster_addr": "10.20.0.12:6808\/176856",
"heartbeat_back_addr": "10.20.0.12:6811\/176856",
"heartbeat_front_addr": "10.20.0.12:6812\/176856",
"state": [
"exists"
]
},
{
"osd": 52,
"uuid": "035e9a74-d6ce-45d5-ba3c-1178f569f27b",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16134,
"up_from": 17695,
"up_thru": 16072,
"down_at": 17715,
"lost_at": 0,
"public_addr": "10.20.0.12:6836\/176879",
"cluster_addr": "10.20.0.12:6837\/176879",
"heartbeat_back_addr": "10.20.0.12:6838\/176879",
"heartbeat_front_addr": "10.20.0.12:6842\/176879",
"state": [
"exists"
]
},
{
"osd": 53,
"uuid": "808fb005-0ffd-4aba-b6ca-e3c8708af1f5",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 17695,
"last_clean_end": 17716,
"up_from": 17822,
"up_thru": 16117,
"down_at": 17834,
"lost_at": 0,
"public_addr": "10.20.0.12:6846\/182870",
"cluster_addr": "10.20.0.12:6847\/182870",
"heartbeat_back_addr": "10.20.0.12:6848\/182870",
"heartbeat_front_addr": "10.20.0.12:6852\/182870",
"state": [
"exists"
]
},
{
"osd": 54,
"uuid": "b1d374ff-5d1b-4690-915f-8e6f1849ea79",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16139,
"up_from": 17731,
"up_thru": 17731,
"down_at": 17751,
"lost_at": 0,
"public_addr": "10.20.0.12:6807\/178622",
"cluster_addr": "10.20.0.12:6808\/178622",
"heartbeat_back_addr": "10.20.0.12:6811\/178622",
"heartbeat_front_addr": "10.20.0.12:6812\/178622",
"state": [
"exists"
]
},
{
"osd": 55,
"uuid": "f6c90ac8-a30a-4a37-8c85-5270fa8b423e",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16144,
"up_from": 17737,
"up_thru": 16129,
"down_at": 17753,
"lost_at": 0,
"public_addr": "10.20.0.12:6836\/178648",
"cluster_addr": "10.20.0.12:6838\/178648",
"heartbeat_back_addr": "10.20.0.12:6842\/178648",
"heartbeat_front_addr": "10.20.0.12:6846\/178648",
"state": [
"exists"
]
},
{
"osd": 56,
"uuid": "6d5a3648-bb8d-4aa7-9686-16885790c5f8",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 17742,
"last_clean_end": 17756,
"up_from": 17820,
"up_thru": 16116,
"down_at": 17836,
"lost_at": 0,
"public_addr": "10.20.0.12:6836\/182573",
"cluster_addr": "10.20.0.12:6837\/182573",
"heartbeat_back_addr": "10.20.0.12:6838\/182573",
"heartbeat_front_addr": "10.20.0.12:6842\/182573",
"state": [
"exists"
]
},
{
"osd": 57,
"uuid": "63338d14-910e-4d77-86c6-bdfde643736b",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16154,
"up_from": 17769,
"up_thru": 16139,
"down_at": 17808,
"lost_at": 0,
"public_addr": "10.20.0.12:6807\/180329",
"cluster_addr": "10.20.0.12:6811\/180329",
"heartbeat_back_addr": "10.20.0.12:6812\/180329",
"heartbeat_front_addr": "10.20.0.12:6836\/180329",
"state": [
"exists"
]
},
{
"osd": 58,
"uuid": "13199a09-0db2-4ac5-9d58-3d633b93d14f",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15783,
"last_clean_end": 16159,
"up_from": 17772,
"up_thru": 17777,
"down_at": 17810,
"lost_at": 0,
"public_addr": "10.20.0.12:6837\/180352",
"cluster_addr": "10.20.0.12:6842\/180352",
"heartbeat_back_addr": "10.20.0.12:6846\/180352",
"heartbeat_front_addr": "10.20.0.12:6847\/180352",
"state": [
"exists"
]
},
{
"osd": 59,
"uuid": "15723089-c846-4d6d-bf23-ee9ea418ba21",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 16566,
"last_clean_end": 17682,
"up_from": 17826,
"up_thru": 17353,
"down_at": 17831,
"lost_at": 0,
"public_addr": "10.20.0.13:6807\/416922",
"cluster_addr": "10.20.0.13:6808\/416922",
"heartbeat_back_addr": "10.20.0.13:6809\/416922",
"heartbeat_front_addr": "10.20.0.13:6811\/416922",
"state": [
"exists"
]
},
{
"osd": 60,
"uuid": "7d768064-4ab0-4962-a95e-1257ffa839eb",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15683,
"last_clean_end": 16131,
"up_from": 17662,
"up_thru": 16066,
"down_at": 17685,
"lost_at": 0,
"public_addr": "10.20.0.13:6807\/406378",
"cluster_addr": "10.20.0.13:6808\/406378",
"heartbeat_back_addr": "10.20.0.13:6809\/406378",
"heartbeat_front_addr": "10.20.0.13:6811\/406378",
"state": [
"exists"
]
},
{
"osd": 61,
"uuid": "576d436b-19d2-4eb1-ae6e-0f19a52c78e6",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 17676,
"last_clean_end": 17685,
"up_from": 17823,
"up_thru": 16080,
"down_at": 17832,
"lost_at": 0,
"public_addr": "10.20.0.13:6802\/416491",
"cluster_addr": "10.20.0.13:6804\/416491",
"heartbeat_back_addr": "10.20.0.13:6805\/416491",
"heartbeat_front_addr": "10.20.0.13:6806\/416491",
"state": [
"exists"
]
},
{
"osd": 62,
"uuid": "5d1a9351-a39d-43c3-8945-6bd551ecb988",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15682,
"last_clean_end": 16133,
"up_from": 17693,
"up_thru": 16121,
"down_at": 17719,
"lost_at": 0,
"public_addr": "10.20.0.13:6802\/407884",
"cluster_addr": "10.20.0.13:6804\/407884",
"heartbeat_back_addr": "10.20.0.13:6805\/407884",
"heartbeat_front_addr": "10.20.0.13:6806\/407884",
"state": [
"exists"
]
},
{
"osd": 63,
"uuid": "ad840dee-a8f7-4074-9f53-729075712b62",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15679,
"last_clean_end": 16141,
"up_from": 17698,
"up_thru": 16079,
"down_at": 17720,
"lost_at": 0,
"public_addr": "10.20.0.13:6807\/407910",
"cluster_addr": "10.20.0.13:6808\/407910",
"heartbeat_back_addr": "10.20.0.13:6809\/407910",
"heartbeat_front_addr": "10.20.0.13:6811\/407910",
"state": [
"exists"
]
},
{
"osd": 64,
"uuid": "a8b2d98a-3be7-4bfe-9187-e1fe77a2b179",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15679,
"last_clean_end": 16143,
"up_from": 17697,
"up_thru": 16140,
"down_at": 17721,
"lost_at": 0,
"public_addr": "10.20.0.13:6812\/407963",
"cluster_addr": "10.20.0.13:6813\/407963",
"heartbeat_back_addr": "10.20.0.13:6814\/407963",
"heartbeat_front_addr": "10.20.0.13:6816\/407963",
"state": [
"exists"
]
},
{
"osd": 65,
"uuid": "eab0c4b3-bc56-4157-b60a-51f7b7ac4732",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15675,
"last_clean_end": 16148,
"up_from": 17725,
"up_thru": 16125,
"down_at": 17747,
"lost_at": 0,
"public_addr": "10.20.0.13:6802\/410040",
"cluster_addr": "10.20.0.13:6805\/410040",
"heartbeat_back_addr": "10.20.0.13:6806\/410040",
"heartbeat_front_addr": "10.20.0.13:6807\/410040",
"state": [
"exists"
]
},
{
"osd": 66,
"uuid": "227736ac-0a48-4b7a-936b-1a9d241c3c99",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15675,
"last_clean_end": 16153,
"up_from": 17727,
"up_thru": 16128,
"down_at": 17748,
"lost_at": 0,
"public_addr": "10.20.0.13:6809\/410063",
"cluster_addr": "10.20.0.13:6811\/410063",
"heartbeat_back_addr": "10.20.0.13:6812\/410063",
"heartbeat_front_addr": "10.20.0.13:6814\/410063",
"state": [
"exists"
]
},
{
"osd": 67,
"uuid": "a8c1b84a-ef5a-47ff-ad49-0c3316b97bf6",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15675,
"last_clean_end": 16158,
"up_from": 17723,
"up_thru": 16089,
"down_at": 17749,
"lost_at": 0,
"public_addr": "10.20.0.13:6816\/410115",
"cluster_addr": "10.20.0.13:6817\/410115",
"heartbeat_back_addr": "10.20.0.13:6818\/410115",
"heartbeat_front_addr": "10.20.0.13:6819\/410115",
"state": [
"exists"
]
},
{
"osd": 68,
"uuid": "0d9a5e37-cc79-46a7-89f6-b5e489dfb23e",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15678,
"last_clean_end": 16162,
"up_from": 17760,
"up_thru": 16070,
"down_at": 17811,
"lost_at": 0,
"public_addr": "10.20.0.13:6802\/412386",
"cluster_addr": "10.20.0.13:6804\/412386",
"heartbeat_back_addr": "10.20.0.13:6805\/412386",
"heartbeat_front_addr": "10.20.0.13:6806\/412386",
"state": [
"exists"
]
},
{
"osd": 69,
"uuid": "ad69308b-7562-4880-b73c-ca05a2d8fc2e",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15678,
"last_clean_end": 16167,
"up_from": 17759,
"up_thru": 16072,
"down_at": 17812,
"lost_at": 0,
"public_addr": "10.20.0.13:6807\/412409",
"cluster_addr": "10.20.0.13:6808\/412409",
"heartbeat_back_addr": "10.20.0.13:6809\/412409",
"heartbeat_front_addr": "10.20.0.13:6812\/412409",
"state": [
"exists"
]
},
{
"osd": 70,
"uuid": "d4b7ba97-3298-4fd0-9b4a-e461e88352de",
"up": 0,
"in": 0,
"weight": 0.000000,
"primary_affinity": 1.000000,
"last_clean_begin": 15674,
"last_clean_end": 16172,
"up_from": 17761,
"up_thru": 17768,
"down_at": 17813,
"lost_at": 0,
"public_addr": "10.20.0.13:6813\/412461",
"cluster_addr": "10.20.0.13:6814\/412461",
"heartbeat_back_addr": "10.20.0.13:6816\/412461",
"heartbeat_front_addr": "10.20.0.13:6817\/412461",
"state": [
"exists"
]
}
],
"osd_xinfo": [
{
"osd": 0,
"down_stamp": "2015-04-26 02:00:19.572511",
"laggy_probability": 0.927101,
"laggy_interval": 46,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 1,
"down_stamp": "2015-04-26 02:00:19.572511",
"laggy_probability": 0.896128,
"laggy_interval": 43,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 2,
"down_stamp": "2015-04-26 02:00:19.572511",
"laggy_probability": 0.882482,
"laggy_interval": 68,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 3,
"down_stamp": "2015-04-26 02:21:39.868625",
"laggy_probability": 0.943535,
"laggy_interval": 41,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 4,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.917737,
"laggy_interval": 56,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 5,
"down_stamp": "2015-04-26 02:02:17.842800",
"laggy_probability": 0.882482,
"laggy_interval": 15,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 6,
"down_stamp": "2015-04-26 02:00:19.572511",
"laggy_probability": 0.917737,
"laggy_interval": 59,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 7,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.917737,
"laggy_interval": 95,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 8,
"down_stamp": "2015-04-26 04:36:29.410592",
"laggy_probability": 0.642416,
"laggy_interval": 58,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 9,
"down_stamp": "2015-04-26 02:00:19.572511",
"laggy_probability": 0.919336,
"laggy_interval": 24,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 10,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.880998,
"laggy_interval": 69,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 11,
"down_stamp": "2015-04-26 04:36:35.768784",
"laggy_probability": 0.604091,
"laggy_interval": 32,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 12,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.849601,
"laggy_interval": 59,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 13,
"down_stamp": "2015-04-26 03:22:55.024663",
"laggy_probability": 0.859651,
"laggy_interval": 42,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 14,
"down_stamp": "2015-04-26 03:47:02.193069",
"laggy_probability": 0.948316,
"laggy_interval": 30,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 15,
"down_stamp": "2015-04-26 03:22:33.512773",
"laggy_probability": 0.945507,
"laggy_interval": 26,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 16,
"down_stamp": "2015-04-26 03:22:20.137195",
"laggy_probability": 0.948413,
"laggy_interval": 19,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 17,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.773384,
"laggy_interval": 72,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 18,
"down_stamp": "2015-04-26 03:22:45.016578",
"laggy_probability": 0.889015,
"laggy_interval": 50,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 19,
"down_stamp": "2015-04-26 03:22:33.512773",
"laggy_probability": 0.931228,
"laggy_interval": 29,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 20,
"down_stamp": "2015-04-26 03:22:55.024663",
"laggy_probability": 0.894947,
"laggy_interval": 43,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 21,
"down_stamp": "2015-04-26 03:22:41.016805",
"laggy_probability": 0.933753,
"laggy_interval": 14,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 22,
"down_stamp": "2015-04-26 03:47:12.044546",
"laggy_probability": 0.951671,
"laggy_interval": 18,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 23,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.849840,
"laggy_interval": 65,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 24,
"down_stamp": "2015-04-26 02:03:26.390577",
"laggy_probability": 0.806616,
"laggy_interval": 47,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 25,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.810979,
"laggy_interval": 52,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 26,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.858681,
"laggy_interval": 132,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 27,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.796432,
"laggy_interval": 70,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 28,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.908335,
"laggy_interval": 39,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 29,
"down_stamp": "2015-04-26 02:03:26.390577",
"laggy_probability": 0.732148,
"laggy_interval": 53,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 30,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.866609,
"laggy_interval": 44,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 31,
"down_stamp": "2015-04-26 02:21:39.868625",
"laggy_probability": 0.866052,
"laggy_interval": 46,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 32,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.812573,
"laggy_interval": 53,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 33,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.797734,
"laggy_interval": 62,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 34,
"down_stamp": "2015-04-26 02:21:21.502025",
"laggy_probability": 0.681026,
"laggy_interval": 108,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 35,
"down_stamp": "2015-04-26 02:03:26.390577",
"laggy_probability": 0.703038,
"laggy_interval": 47,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 36,
"down_stamp": "2015-04-26 04:26:34.625477",
"laggy_probability": 0.583865,
"laggy_interval": 14,
"features": 1125899906842623,
"old_weight": 0
},
{
"osd": 37,
"down_stamp": "2015-04-26 04:26:37.902343",
"laggy_probability": 0.585348,
"laggy_interval": 35,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 38,
"down_stamp": "2015-04-26 04:26:41.237620",
"laggy_probability": 0.561149,
"laggy_interval": 37,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 39,
"down_stamp": "2015-04-26 04:41:21.591738",
"laggy_probability": 0.433094,
"laggy_interval": 29,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 40,
"down_stamp": "2015-04-26 04:31:10.523987",
"laggy_probability": 0.641786,
"laggy_interval": 19,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 41,
"down_stamp": "2015-04-26 04:31:13.837051",
"laggy_probability": 0.581195,
"laggy_interval": 32,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 42,
"down_stamp": "2015-04-26 04:34:14.780969",
"laggy_probability": 0.618319,
"laggy_interval": 45,
"features": 1125899906842623,
"old_weight": 0
},
{
"osd": 43,
"down_stamp": "2015-04-26 04:34:17.897798",
"laggy_probability": 0.618319,
"laggy_interval": 14,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 44,
"down_stamp": "2015-04-26 04:34:20.994697",
"laggy_probability": 0.617709,
"laggy_interval": 37,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 45,
"down_stamp": "2015-04-26 04:41:24.759914",
"laggy_probability": 0.318740,
"laggy_interval": 45,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 46,
"down_stamp": "2015-04-26 04:37:59.919908",
"laggy_probability": 0.583313,
"laggy_interval": 62,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 47,
"down_stamp": "2015-04-26 04:41:27.882318",
"laggy_probability": 0.368083,
"laggy_interval": 24,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 48,
"down_stamp": "2015-04-26 04:41:16.379505",
"laggy_probability": 0.376217,
"laggy_interval": 36,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 49,
"down_stamp": "2015-04-26 04:27:03.567206",
"laggy_probability": 0.592149,
"laggy_interval": 25,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 50,
"down_stamp": "2015-04-26 04:27:04.676795",
"laggy_probability": 0.555430,
"laggy_interval": 43,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 51,
"down_stamp": "2015-04-26 04:31:16.034637",
"laggy_probability": 0.496930,
"laggy_interval": 21,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 52,
"down_stamp": "2015-04-26 04:31:19.128934",
"laggy_probability": 0.587214,
"laggy_interval": 22,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 53,
"down_stamp": "2015-04-26 04:41:19.500570",
"laggy_probability": 0.413507,
"laggy_interval": 16,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 54,
"down_stamp": "2015-04-26 04:34:04.259736",
"laggy_probability": 0.584793,
"laggy_interval": 19,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 55,
"down_stamp": "2015-04-26 04:34:07.571042",
"laggy_probability": 0.555430,
"laggy_interval": 20,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 56,
"down_stamp": "2015-04-26 04:41:22.650530",
"laggy_probability": 0.382869,
"laggy_interval": 13,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 57,
"down_stamp": "2015-04-26 04:38:03.048749",
"laggy_probability": 0.490565,
"laggy_interval": 32,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 58,
"down_stamp": "2015-04-26 04:38:06.220080",
"laggy_probability": 0.546174,
"laggy_interval": 46,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 59,
"down_stamp": "2015-04-26 04:41:11.190236",
"laggy_probability": 0.306253,
"laggy_interval": 14,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 60,
"down_stamp": "2015-04-26 04:27:18.492423",
"laggy_probability": 0.554090,
"laggy_interval": 17,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 61,
"down_stamp": "2015-04-26 04:41:14.304955",
"laggy_probability": 0.417258,
"laggy_interval": 12,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 62,
"down_stamp": "2015-04-26 04:31:35.273855",
"laggy_probability": 0.597863,
"laggy_interval": 12,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 63,
"down_stamp": "2015-04-26 04:31:36.357348",
"laggy_probability": 0.545616,
"laggy_interval": 17,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 64,
"down_stamp": "2015-04-26 04:31:39.476062",
"laggy_probability": 0.554090,
"laggy_interval": 31,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 65,
"down_stamp": "2015-04-26 04:33:52.803421",
"laggy_probability": 0.554090,
"laggy_interval": 32,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 66,
"down_stamp": "2015-04-26 04:33:55.914944",
"laggy_probability": 0.554090,
"laggy_interval": 48,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 67,
"down_stamp": "2015-04-26 04:33:59.049520",
"laggy_probability": 0.597863,
"laggy_interval": 19,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 68,
"down_stamp": "2015-04-26 04:38:11.822916",
"laggy_probability": 0.554090,
"laggy_interval": 16,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 69,
"down_stamp": "2015-04-26 04:38:14.921856",
"laggy_probability": 0.554090,
"laggy_interval": 22,
"features": 1125899906842623,
"old_weight": 65536
},
{
"osd": 70,
"down_stamp": "2015-04-26 04:38:18.015390",
"laggy_probability": 0.554090,
"laggy_interval": 35,
"features": 1125899906842623,
"old_weight": 65536
}
],
"pg_temp": [],
"primary_temp": [],
"blacklist": [],
"erasure_code_profiles": {
"default": {
"directory": "\/usr\/lib\/ceph\/erasure-code",
"k": "2",
"m": "1",
"plugin": "jerasure",
"technique": "reed_sol_van"
}
}
}
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
