Re: Upgrade from Giant to Hammer and after some basic operations most of the OSD's went down

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi

Here you go

Br,
Tuomas



-----Original Message-----
From: Sage Weil [mailto:sweil@xxxxxxxxxx] 
Sent: 27. huhtikuuta 2015 19:23
To: Tuomas Juntunen
Cc: 'Samuel Just'; ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Upgrade from Giant to Hammer and after some basic
operations most of the OSD's went down

On Mon, 27 Apr 2015, Tuomas Juntunen wrote:
> Thanks for the info.
> 
> For my knowledge there was no snapshots on that pool, but cannot 
> verify that.

Can you attach a 'ceph osd dump -f json-pretty'?  That will shed a bit more
light on what happened (and the simplest way to fix it).

sage


> Any way to make this work again? Removing the tier and other settings 
> didn't fix it, I tried it the second this happened.
> 
> Br,
> Tuomas
> 
> -----Original Message-----
> From: Samuel Just [mailto:sjust@xxxxxxxxxx]
> Sent: 27. huhtikuuta 2015 15:50
> To: tuomas juntunen
> Cc: ceph-users@xxxxxxxxxxxxxx
> Subject: Re:  Upgrade from Giant to Hammer and after some 
> basic operations most of the OSD's went down
> 
> So, the base tier is what determines the snapshots for the cache/base pool
amalgam.  You added a populated pool complete with snapshots on top of a
base tier without snapshots.  Apparently, it caused an existential crisis
for the snapshot code.  That's one of the reasons why there is a
--force-nonempty flag for that operation, I think.  I think the immediate
answer is probably to disallow pools with snapshots as a cache tier
altogether until we think of a good way to make it work.
> -Sam
> 
> ----- Original Message -----
> From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> To: "Samuel Just" <sjust@xxxxxxxxxx>
> Cc: ceph-users@xxxxxxxxxxxxxx
> Sent: Monday, April 27, 2015 4:56:58 AM
> Subject: Re:  Upgrade from Giant to Hammer and after some 
> basic operations most of the OSD's went down
> 
> 
> 
> The following:
> 
> ceph osd tier add img images --force-nonempty ceph osd tier cache-mode 
> images forward ceph osd tier set-overlay img images
> 
> Idea was to make images as a tier to img, move data to img then change
clients to use the new img pool.
> 
> Br,
> Tuomas
> 
> > Can you explain exactly what you mean by:
> >
> > "Also I created one pool for tier to be able to move data without
outage."
> >
> > -Sam
> > ----- Original Message -----
> > From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> > To: "Ian Colle" <icolle@xxxxxxxxxx>
> > Cc: ceph-users@xxxxxxxxxxxxxx
> > Sent: Monday, April 27, 2015 4:23:44 AM
> > Subject: Re:  Upgrade from Giant to Hammer and after 
> > some basic operations most of the OSD's went down
> >
> > Hi
> >
> > Any solution for this yet?
> >
> > Br,
> > Tuomas
> >
> >> It looks like you may have hit http://tracker.ceph.com/issues/7915
> >>
> >> Ian R. Colle
> >> Global Director
> >> of Software Engineering
> >> Red Hat (Inktank is now part of Red Hat!) 
> >> http://www.linkedin.com/in/ircolle
> >> http://www.twitter.com/ircolle
> >> Cell: +1.303.601.7713
> >> Email: icolle@xxxxxxxxxx
> >>
> >> ----- Original Message -----
> >> From: "tuomas juntunen" <tuomas.juntunen@xxxxxxxxxxxxxxx>
> >> To: ceph-users@xxxxxxxxxxxxxx
> >> Sent: Monday, April 27, 2015 1:56:29 PM
> >> Subject:  Upgrade from Giant to Hammer and after some 
> >> basic operations most of the OSD's went down
> >>
> >>
> >>
> >> I upgraded Ceph from 0.87 Giant to 0.94.1 Hammer
> >>
> >> Then created new pools and deleted some old ones. Also I created 
> >> one pool for tier to be able to move data without outage.
> >>
> >> After these operations all but 10 OSD's are down and creating this 
> >> kind of messages to logs, I get more than 100gb of these in a night:
> >>
> >>  -19> 2015-04-27 10:17:08.808584 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 ec=1 
> >> les/c
> >> 16609/16659
> >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 pi=15659-16589/42
> >> crt=8480'7 lcod
> >> 0'0 inactive NOTIFY] enter Started
> >>    -18> 2015-04-27 10:17:08.808596 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 ec=1 
> >> les/c
> >> 16609/16659
> >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 pi=15659-16589/42
> >> crt=8480'7 lcod
> >> 0'0 inactive NOTIFY] enter Start
> >>    -17> 2015-04-27 10:17:08.808608 7fd8e748d700  1 osd.23 pg_epoch: 
> >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 ec=1 
> >> les/c
> >> 16609/16659
> >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 pi=15659-16589/42
> >> crt=8480'7 lcod
> >> 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
> >>    -16> 2015-04-27 10:17:08.808621 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 ec=1 
> >> les/c
> >> 16609/16659
> >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 pi=15659-16589/42
> >> crt=8480'7 lcod
> >> 0'0 inactive NOTIFY] exit Start 0.000025 0 0.000000
> >>    -15> 2015-04-27 10:17:08.808637 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[0.189( v 8480'7 (0'0,8480'7] local-les=16609 n=0 ec=1 
> >> les/c
> >> 16609/16659
> >> 16590/16590/16590) [24,3,23] r=2 lpr=17838 pi=15659-16589/42
> >> crt=8480'7 lcod
> >> 0'0 inactive NOTIFY] enter Started/Stray
> >>    -14> 2015-04-27 10:17:08.808796 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c 
> >> 17879/17879
> >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive NOTIFY] 
> >> exit Reset 0.119467 4 0.000037
> >>    -13> 2015-04-27 10:17:08.808817 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c 
> >> 17879/17879
> >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive NOTIFY] 
> >> enter Started
> >>    -12> 2015-04-27 10:17:08.808828 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c 
> >> 17879/17879
> >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive NOTIFY] 
> >> enter Start
> >>    -11> 2015-04-27 10:17:08.808838 7fd8e748d700  1 osd.23 pg_epoch: 
> >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c 
> >> 17879/17879
> >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive NOTIFY]
> >> state<Start>: transitioning to Stray
> >>    -10> 2015-04-27 10:17:08.808849 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c 
> >> 17879/17879
> >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive NOTIFY] 
> >> exit Start 0.000020 0 0.000000
> >>     -9> 2015-04-27 10:17:08.808861 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[10.181( empty local-les=17879 n=0 ec=17863 les/c 
> >> 17879/17879
> >> 17863/17863/17863) [25,5,23] r=2 lpr=17879 crt=0'0 inactive NOTIFY] 
> >> enter Started/Stray
> >>     -8> 2015-04-27 10:17:08.809427 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive] 
> >> exit Reset 7.511623 45 0.000165
> >>     -7> 2015-04-27 10:17:08.809445 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive] 
> >> enter Started
> >>     -6> 2015-04-27 10:17:08.809456 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive] 
> >> enter Start
> >>     -5> 2015-04-27 10:17:08.809468 7fd8e748d700  1 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive]
> >> state<Start>: transitioning to Primary
> >>     -4> 2015-04-27 10:17:08.809479 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive] 
> >> exit Start 0.000023 0 0.000000
> >>     -3> 2015-04-27 10:17:08.809492 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive] 
> >> enter Started/Primary
> >>     -2> 2015-04-27 10:17:08.809502 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 inactive] 
> >> enter Started/Primary/Peering
> >>     -1> 2015-04-27 10:17:08.809513 7fd8e748d700  5 osd.23 pg_epoch: 
> >> 17882 pg[2.189( empty local-les=16127 n=0 ec=1 les/c 16127/16344
> >> 16125/16125/16125) [23,5] r=0 lpr=17838 crt=0'0 mlcod 0'0 peering] 
> >> enter Started/Primary/Peering/GetInfo
> >>      0> 2015-04-27 10:17:08.813837 7fd8e748d700 -1
./include/interval_set.h:
> >> In
> >> function 'void interval_set<T>::erase(T, T) [with T = snapid_t]' 
> >> thread
> >> 7fd8e748d700 time 2015-04-27 10:17:08.809899
> >> ./include/interval_set.h: 385: FAILED assert(_size >= 0)
> >>
> >>  ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
> >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >> const*)+0x8b)
> >> [0xbc271b]
> >>  2: (interval_set<snapid_t>::subtract(interval_set<snapid_t>
> >> const&)+0xb0) [0x82cd50]
> >>  3: (PGPool::update(std::tr1::shared_ptr<OSDMap const>)+0x52e) 
> >> [0x80113e]
> >>  4: (PG::handle_advance_map(std::tr1::shared_ptr<OSDMap const>, 
> >> std::tr1::shared_ptr<OSDMap const>, std::vector<int, 
> >> std::allocator<int> >&, int, std::vector<int, std::allocator<int> 
> >> >&, int, PG::RecoveryCtx*)+0x282) [0x801652]
> >>  5: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, 
> >> PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>,
> >> std::less<boost::intrusive_ptr<PG> >, 
> >> std::allocator<boost::intrusive_ptr<PG> > >*)+0x2c3) [0x6b0e43]
> >>  6: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> 
> >> > const&,
> >> ThreadPool::TPHandle&)+0x21c) [0x6b191c]
> >>  7: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > 
> >> const&,
> >> ThreadPool::TPHandle&)+0x18) [0x709278]
> >>  8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbb38ae]
> >>  9: (ThreadPool::WorkThread::entry()+0x10) [0xbb4950]
> >>  10: (()+0x8182) [0x7fd906946182]
> >>  11: (clone()+0x6d) [0x7fd904eb147d]
> >>
> >> Also by monitoring (ceph -w) I get the following messages, also lots of
them.
> >>
> >> 2015-04-27 10:39:52.935812 mon.0 [INF] from='client.?
10.20.0.13:0/1174409'
> >> entity='osd.30' cmd=[{"prefix": "osd crush create-or-move", "args":
> >> ["host=ceph3", "root=default"], "id": 30, "weight": 1.82}]: 
> >> dispatch
> >> 2015-04-27 10:39:53.297376 mon.0 [INF] from='client.?
10.20.0.13:0/1174483'
> >> entity='osd.26' cmd=[{"prefix": "osd crush create-or-move", "args":
> >> ["host=ceph3", "root=default"], "id": 26, "weight": 1.82}]: 
> >> dispatch
> >>
> >>
> >> This is a cluster of 3 nodes with 36 OSD's, nodes are also mons and 
> >> mds's to save servers. All run Ubuntu 14.04.2.
> >>
> >> I have pretty much tried everything I could think of.
> >>
> >> Restarting daemons doesn't help.
> >>
> >> Any help would be appreciated. I can also provide more logs if 
> >> necessary. They just seem to get pretty large in few moments.
> >>
> >> Thank you
> >> Tuomas
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@xxxxxxxxxxxxxx
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >>
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
{
    "epoch": 18128,
    "fsid": "a2974742-3805-4cd3-bc79-765f2bddaefe",
    "created": "2014-10-15 20:43:45.186949",
    "modified": "2015-04-27 17:00:13.746815",
    "flags": "",
    "cluster_snapshot": "",
    "pool_max": 17,
    "max_osd": 71,
    "pools": [
        {
            "pool": 0,
            "pool_name": "data",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 45,
            "last_change": "1112",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 1,
            "pool_name": "metadata",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1114",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 2,
            "pool_name": "rbd",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 2,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1116",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 3,
            "pool_name": "volumes",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "9974",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 23,
            "snap_epoch": 9974,
            "pool_snaps": [],
            "removed_snaps": "[1~17]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 4,
            "pool_name": "images",
            "flags": 9,
            "flags_names": "hashpspool,incomplete_clones",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "17905",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 17882,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 0,
            "cache_target_full_ratio_micro": 0,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 6,
            "pool_name": "vms",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "1122",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "default",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 1,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 7,
            "pool_name": "san",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 1,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 4096,
            "pg_placement_num": 4096,
            "crash_replay_interval": 0,
            "last_change": "14096",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 8,
            "pool_name": "vol-ssd-accelerated",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 2,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 1024,
            "pg_placement_num": 1024,
            "crash_replay_interval": 0,
            "last_change": "17861",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 14,
            "pool_name": "backup",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 2,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 128,
            "pg_placement_num": 128,
            "crash_replay_interval": 0,
            "last_change": "18018",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 15,
            "pool_name": "img",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 2,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 256,
            "pg_placement_num": 256,
            "crash_replay_interval": 0,
            "last_change": "18019",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 16,
            "pool_name": "vm",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 2,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 1024,
            "pg_placement_num": 1024,
            "crash_replay_interval": 0,
            "last_change": "18020",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        },
        {
            "pool": 17,
            "pool_name": "infradisks",
            "flags": 1,
            "flags_names": "hashpspool",
            "type": 1,
            "size": 3,
            "min_size": 2,
            "crush_ruleset": 0,
            "object_hash": 2,
            "pg_num": 256,
            "pg_placement_num": 256,
            "crash_replay_interval": 0,
            "last_change": "18021",
            "last_force_op_resend": "0",
            "auid": 0,
            "snap_mode": "selfmanaged",
            "snap_seq": 0,
            "snap_epoch": 0,
            "pool_snaps": [],
            "removed_snaps": "[]",
            "quota_max_bytes": 0,
            "quota_max_objects": 0,
            "tiers": [],
            "tier_of": -1,
            "read_tier": -1,
            "write_tier": -1,
            "cache_mode": "none",
            "target_max_bytes": 0,
            "target_max_objects": 0,
            "cache_target_dirty_ratio_micro": 400000,
            "cache_target_full_ratio_micro": 800000,
            "cache_min_flush_age": 0,
            "cache_min_evict_age": 0,
            "erasure_code_profile": "",
            "hit_set_params": {
                "type": "none"
            },
            "hit_set_period": 0,
            "hit_set_count": 0,
            "min_read_recency_for_promote": 0,
            "stripe_width": 0,
            "expected_num_objects": 0
        }
    ],
    "osds": [
        {
            "osd": 0,
            "uuid": "757c3bc5-4d00-4344-8de4-82f5379c96af",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15738,
            "last_clean_end": 16079,
            "up_from": 16081,
            "up_thru": 17872,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6981\/245797",
            "cluster_addr": "10.20.0.11:6881\/5245797",
            "heartbeat_back_addr": "10.20.0.11:6883\/5245797",
            "heartbeat_front_addr": "10.20.0.11:6887\/5245797",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 1,
            "uuid": "c7eaa4ac-99fc-46db-84aa-a67274896ec8",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15740,
            "last_clean_end": 16079,
            "up_from": 16083,
            "up_thru": 17873,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6813\/245273",
            "cluster_addr": "10.20.0.11:6888\/4245273",
            "heartbeat_back_addr": "10.20.0.11:6890\/4245273",
            "heartbeat_front_addr": "10.20.0.11:6894\/4245273",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 2,
            "uuid": "206b2949-4adf-4789-8e06-f68a8ee819c9",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15739,
            "last_clean_end": 16079,
            "up_from": 16081,
            "up_thru": 17874,
            "down_at": 17915,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6808\/245269",
            "cluster_addr": "10.20.0.11:6811\/4245269",
            "heartbeat_back_addr": "10.20.0.11:6823\/4245269",
            "heartbeat_front_addr": "10.20.0.11:6831\/4245269",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 3,
            "uuid": "90b7c219-4dcd-48ea-a24d-f3b796a521e4",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15736,
            "last_clean_end": 16125,
            "up_from": 16128,
            "up_thru": 17873,
            "down_at": 17920,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6848\/245357",
            "cluster_addr": "10.20.0.11:6886\/6245357",
            "heartbeat_back_addr": "10.20.0.11:6891\/6245357",
            "heartbeat_front_addr": "10.20.0.11:6892\/6245357",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 4,
            "uuid": "049ef94f-121a-4e71-8ba6-27eaebf0a569",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15737,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17871,
            "down_at": 17925,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6854\/245375",
            "cluster_addr": "10.20.0.11:6849\/5245375",
            "heartbeat_back_addr": "10.20.0.11:6850\/5245375",
            "heartbeat_front_addr": "10.20.0.11:6853\/5245375",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 5,
            "uuid": "2437a53b-339e-45af-b4de-0fc675d27405",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15734,
            "last_clean_end": 16087,
            "up_from": 16089,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6834\/245323",
            "cluster_addr": "10.20.0.11:6803\/4245323",
            "heartbeat_back_addr": "10.20.0.11:6804\/4245323",
            "heartbeat_front_addr": "10.20.0.11:6805\/4245323",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 6,
            "uuid": "f117ceed-b1fd-4069-99fe-b7aba9f3ef8d",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15738,
            "last_clean_end": 16079,
            "up_from": 16083,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6859\/245401",
            "cluster_addr": "10.20.0.11:6865\/5245401",
            "heartbeat_back_addr": "10.20.0.11:6868\/5245401",
            "heartbeat_front_addr": "10.20.0.11:6874\/5245401",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 7,
            "uuid": "e98e9b8a-9c62-4e3e-bdb4-c2c30103c0c1",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15730,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17872,
            "down_at": 17925,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6801\/245246",
            "cluster_addr": "10.20.0.11:6812\/5245246",
            "heartbeat_back_addr": "10.20.0.11:6814\/5245246",
            "heartbeat_front_addr": "10.20.0.11:6821\/5245246",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 8,
            "uuid": "41e471cd-fafe-4422-8bf5-22018bbe1375",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15736,
            "last_clean_end": 17783,
            "up_from": 17795,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6810\/458698",
            "cluster_addr": "10.20.0.11:6819\/458698",
            "heartbeat_back_addr": "10.20.0.11:6826\/458698",
            "heartbeat_front_addr": "10.20.0.11:6833\/458698",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 9,
            "uuid": "d68eeebd-d058-4b1c-a30a-994bf8fc8030",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15733,
            "last_clean_end": 16079,
            "up_from": 16081,
            "up_thru": 17873,
            "down_at": 17930,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6969\/245600",
            "cluster_addr": "10.20.0.11:6847\/5245600",
            "heartbeat_back_addr": "10.20.0.11:6860\/5245600",
            "heartbeat_front_addr": "10.20.0.11:6864\/5245600",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 10,
            "uuid": "660747d6-3f47-449a-bc69-5399b0d54ff6",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15736,
            "last_clean_end": 16124,
            "up_from": 16128,
            "up_thru": 17875,
            "down_at": 17935,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6822\/245281",
            "cluster_addr": "10.20.0.11:6855\/4245281",
            "heartbeat_back_addr": "10.20.0.11:6856\/4245281",
            "heartbeat_front_addr": "10.20.0.11:6866\/4245281",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 11,
            "uuid": "805965b1-127f-44a6-9a05-8a643eb7a512",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15737,
            "last_clean_end": 17786,
            "up_from": 17801,
            "up_thru": 17870,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.11:6835\/458813",
            "cluster_addr": "10.20.0.11:6836\/458813",
            "heartbeat_back_addr": "10.20.0.11:6837\/458813",
            "heartbeat_front_addr": "10.20.0.11:6838\/458813",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 12,
            "uuid": "61fbfcbe-d642-478f-9620-f9d72ee96238",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16126,
            "up_from": 16129,
            "up_thru": 17872,
            "down_at": 17935,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6873\/7542",
            "cluster_addr": "10.20.0.12:6947\/4007542",
            "heartbeat_back_addr": "10.20.0.12:6948\/4007542",
            "heartbeat_front_addr": "10.20.0.12:6949\/4007542",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 13,
            "uuid": "6faad33b-00be-42a4-92ba-08be5ab7f995",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16705,
            "up_from": 16707,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6821\/7274",
            "cluster_addr": "10.20.0.12:6859\/5007274",
            "heartbeat_back_addr": "10.20.0.12:6860\/5007274",
            "heartbeat_front_addr": "10.20.0.12:6861\/5007274",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 14,
            "uuid": "f301705b-e725-443d-96e1-d9ec9aafe657",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16962,
            "up_from": 16966,
            "up_thru": 17871,
            "down_at": 17940,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6809\/7209",
            "cluster_addr": "10.20.0.12:6803\/7007209",
            "heartbeat_back_addr": "10.20.0.12:6804\/7007209",
            "heartbeat_front_addr": "10.20.0.12:6810\/7007209",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 15,
            "uuid": "536bf483-10de-44b0-8e1e-4f349fbe572a",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15785,
            "last_clean_end": 16700,
            "up_from": 16715,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6881\/7613",
            "cluster_addr": "10.20.0.12:6845\/7007613",
            "heartbeat_back_addr": "10.20.0.12:6849\/7007613",
            "heartbeat_front_addr": "10.20.0.12:6850\/7007613",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 16,
            "uuid": "4185bd20-8eb0-4616-b36e-bacb181ae40e",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16688,
            "up_from": 16693,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6813\/7210",
            "cluster_addr": "10.20.0.12:6814\/7007210",
            "heartbeat_back_addr": "10.20.0.12:6815\/7007210",
            "heartbeat_front_addr": "10.20.0.12:6816\/7007210",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 17,
            "uuid": "a6f2f5b4-477f-48f9-9acf-d5b7a6c88b98",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17872,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6817\/7215",
            "cluster_addr": "10.20.0.12:6840\/3007215",
            "heartbeat_back_addr": "10.20.0.12:6843\/3007215",
            "heartbeat_front_addr": "10.20.0.12:6844\/3007215",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 18,
            "uuid": "b31b0bd8-938a-496d-91bc-19bf4f794f82",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16700,
            "up_from": 16712,
            "up_thru": 17872,
            "down_at": 17942,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6805\/7185",
            "cluster_addr": "10.20.0.12:6829\/5007185",
            "heartbeat_back_addr": "10.20.0.12:6830\/5007185",
            "heartbeat_front_addr": "10.20.0.12:6841\/5007185",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 19,
            "uuid": "d76b6bd5-1ef3-436c-a75d-3587c515eb56",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16693,
            "up_from": 16706,
            "up_thru": 17872,
            "down_at": 16693,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6885\/7622",
            "cluster_addr": "10.20.0.12:6802\/7007622",
            "heartbeat_back_addr": "10.20.0.12:6818\/7007622",
            "heartbeat_front_addr": "10.20.0.12:6819\/7007622",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 20,
            "uuid": "8e4dd982-a4c5-4ca5-9fc5-243f55c4db57",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16703,
            "up_from": 16705,
            "up_thru": 17872,
            "down_at": 16703,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6825\/7316",
            "cluster_addr": "10.20.0.12:6827\/5007316",
            "heartbeat_back_addr": "10.20.0.12:6831\/5007316",
            "heartbeat_front_addr": "10.20.0.12:6851\/5007316",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 21,
            "uuid": "760aaf28-0a34-4bbc-af0c-2654b0a43fff",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16699,
            "up_from": 16701,
            "up_thru": 17870,
            "down_at": 16698,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6801\/7180",
            "cluster_addr": "10.20.0.12:6820\/7007180",
            "heartbeat_back_addr": "10.20.0.12:6822\/7007180",
            "heartbeat_front_addr": "10.20.0.12:6828\/7007180",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 22,
            "uuid": "40322a34-ab31-4760-b71e-a7672f812cb3",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16966,
            "up_from": 16969,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6869\/7509",
            "cluster_addr": "10.20.0.12:6824\/8007509",
            "heartbeat_back_addr": "10.20.0.12:6832\/8007509",
            "heartbeat_front_addr": "10.20.0.12:6834\/8007509",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 23,
            "uuid": "e1d81949-f4b5-4cf2-b6af-dccaaeb30ed7",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15783,
            "last_clean_end": 16123,
            "up_from": 16125,
            "up_thru": 17875,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6877\/7592",
            "cluster_addr": "10.20.0.12:6826\/4007592",
            "heartbeat_back_addr": "10.20.0.12:6835\/4007592",
            "heartbeat_front_addr": "10.20.0.12:6839\/4007592",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 24,
            "uuid": "ede77283-a423-4c6b-9c6e-b0e807c63cb5",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15688,
            "last_clean_end": 16109,
            "up_from": 16112,
            "up_thru": 17871,
            "down_at": 16109,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6921\/213483",
            "cluster_addr": "10.20.0.13:6815\/3213483",
            "heartbeat_back_addr": "10.20.0.13:6841\/3213483",
            "heartbeat_front_addr": "10.20.0.13:6846\/3213483",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 25,
            "uuid": "7cfe85f8-3ae9-493d-9801-025ff6c6265d",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15686,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17871,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6801\/212863",
            "cluster_addr": "10.20.0.13:6918\/3212863",
            "heartbeat_back_addr": "10.20.0.13:6932\/3212863",
            "heartbeat_front_addr": "10.20.0.13:6934\/3212863",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 26,
            "uuid": "266f6d70-519f-4c24-bca2-236495a600a7",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15692,
            "last_clean_end": 16125,
            "up_from": 16130,
            "up_thru": 17872,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6917\/213469",
            "cluster_addr": "10.20.0.13:6896\/4213469",
            "heartbeat_back_addr": "10.20.0.13:6945\/4213469",
            "heartbeat_front_addr": "10.20.0.13:6946\/4213469",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 27,
            "uuid": "68644fa9-9459-4db0-a6c9-01661645038b",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15689,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17871,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6899\/213448",
            "cluster_addr": "10.20.0.13:6876\/3213448",
            "heartbeat_back_addr": "10.20.0.13:6902\/3213448",
            "heartbeat_front_addr": "10.20.0.13:6904\/3213448",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 28,
            "uuid": "fc3d5749-7673-4100-a0d4-f25e9cc0bc88",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15688,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17871,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6892\/213419",
            "cluster_addr": "10.20.0.13:6866\/5213419",
            "heartbeat_back_addr": "10.20.0.13:6867\/5213419",
            "heartbeat_front_addr": "10.20.0.13:6873\/5213419",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 29,
            "uuid": "cb5feda9-de3f-4e42-bb73-7945b4928b22",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15689,
            "last_clean_end": 16109,
            "up_from": 16113,
            "up_thru": 17872,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6937\/213527",
            "cluster_addr": "10.20.0.13:6898\/2213527",
            "heartbeat_back_addr": "10.20.0.13:6926\/2213527",
            "heartbeat_front_addr": "10.20.0.13:6927\/2213527",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 30,
            "uuid": "ef1e65bb-a634-4096-9466-1262af55db01",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15693,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17872,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6910\/213454",
            "cluster_addr": "10.20.0.13:6823\/4213454",
            "heartbeat_back_addr": "10.20.0.13:6838\/4213454",
            "heartbeat_front_addr": "10.20.0.13:6852\/4213454",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 31,
            "uuid": "3dad6393-67a8-43d4-ba8d-ffd320827396",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15688,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6929\/213515",
            "cluster_addr": "10.20.0.13:6909\/4213515",
            "heartbeat_back_addr": "10.20.0.13:6940\/4213515",
            "heartbeat_front_addr": "10.20.0.13:6941\/4213515",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 32,
            "uuid": "db6f3afa-53ed-453a-97e3-861e88cb818f",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15684,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17873,
            "down_at": 16123,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6884\/213261",
            "cluster_addr": "10.20.0.13:6856\/3213261",
            "heartbeat_back_addr": "10.20.0.13:6863\/3213261",
            "heartbeat_front_addr": "10.20.0.13:6865\/3213261",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 33,
            "uuid": "d5e59852-06b4-4a30-8c5e-ff7e328b5455",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15692,
            "last_clean_end": 16125,
            "up_from": 16130,
            "up_thru": 17872,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6900\/213450",
            "cluster_addr": "10.20.0.13:6936\/3213450",
            "heartbeat_back_addr": "10.20.0.13:6938\/3213450",
            "heartbeat_front_addr": "10.20.0.13:6939\/3213450",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 34,
            "uuid": "f35a10c5-217a-4cfb-88b9-7334bda441b8",
            "up": 0,
            "in": 0,
            "weight": 0.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15687,
            "last_clean_end": 16125,
            "up_from": 16127,
            "up_thru": 17871,
            "down_at": 17884,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6831\/212981",
            "cluster_addr": "10.20.0.13:6942\/2212981",
            "heartbeat_back_addr": "10.20.0.13:6943\/2212981",
            "heartbeat_front_addr": "10.20.0.13:6944\/2212981",
            "state": [
                "autoout",
                "exists"
            ]
        },
        {
            "osd": 35,
            "uuid": "335e797f-a390-4f08-9da6-9ab76ffb12ae",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 15687,
            "last_clean_end": 16109,
            "up_from": 16112,
            "up_thru": 17872,
            "down_at": 16109,
            "lost_at": 0,
            "public_addr": "10.20.0.13:6877\/213258",
            "cluster_addr": "10.20.0.13:6803\/2213258",
            "heartbeat_back_addr": "10.20.0.13:6832\/2213258",
            "heartbeat_front_addr": "10.20.0.13:6833\/2213258",
            "state": [
                "exists",
                "up"
            ]
        },
        {
            "osd": 36,
            "uuid": "33c11fa1-1b03-42e4-8296-dc55ba052b35",
            "up": 1,
            "in": 1,
            "weight": 1.000000,
            "primary_affinity": 1.000000,
            "last_clean_begin": 18112,
            "last_clean_end": 18118,
            "up_from": 18121,
            "up_thru": 18121,
            "down_at": 18120,
            "lost_at": 0,
            "public_addr": "10.20.0.12:6809\/3785178",
            "cluster_addr": "10.20.0.12:6810\/3785178",
            "heartbeat_back_addr": "10.20.0.12:6811\/3785178",
            "heartbeat_front_addr": "10.20.0.12:6812\/3785178",
            "state": [
                "exists",
                "up"
            ]
        }
    ],
    "osd_xinfo": [
        {
            "osd": 0,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.927101,
            "laggy_interval": 46,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 1,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.896128,
            "laggy_interval": 43,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 2,
            "down_stamp": "2015-04-26 14:39:36.274440",
            "laggy_probability": 0.882482,
            "laggy_interval": 68,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 3,
            "down_stamp": "2015-04-26 14:39:41.282378",
            "laggy_probability": 0.943535,
            "laggy_interval": 41,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 4,
            "down_stamp": "2015-04-26 14:39:46.308944",
            "laggy_probability": 0.917737,
            "laggy_interval": 56,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 5,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.882482,
            "laggy_interval": 15,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 6,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.917737,
            "laggy_interval": 59,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 7,
            "down_stamp": "2015-04-26 14:39:46.308944",
            "laggy_probability": 0.917737,
            "laggy_interval": 95,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 8,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.642416,
            "laggy_interval": 58,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 9,
            "down_stamp": "2015-04-26 14:39:51.653157",
            "laggy_probability": 0.919336,
            "laggy_interval": 24,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 10,
            "down_stamp": "2015-04-26 14:39:56.661476",
            "laggy_probability": 0.880998,
            "laggy_interval": 69,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 11,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.604091,
            "laggy_interval": 32,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 12,
            "down_stamp": "2015-04-26 14:39:56.661476",
            "laggy_probability": 0.849601,
            "laggy_interval": 59,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 13,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.859651,
            "laggy_interval": 42,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 14,
            "down_stamp": "2015-04-26 14:40:01.673316",
            "laggy_probability": 0.948316,
            "laggy_interval": 30,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 15,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.945507,
            "laggy_interval": 26,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 16,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.948413,
            "laggy_interval": 19,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 17,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.773384,
            "laggy_interval": 72,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 18,
            "down_stamp": "2015-04-26 14:40:06.682693",
            "laggy_probability": 0.889015,
            "laggy_interval": 50,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 19,
            "down_stamp": "2015-04-26 03:22:33.512773",
            "laggy_probability": 0.931228,
            "laggy_interval": 29,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 20,
            "down_stamp": "2015-04-26 03:22:55.024663",
            "laggy_probability": 0.894947,
            "laggy_interval": 43,
            "features": 1125899906842623,
            "old_weight": 0
        },
        {
            "osd": 21,
            "down_stamp": "2015-04-26 03:22:41.016805",
            "laggy_probability": 0.933753,
            "laggy_interval": 14,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 22,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.951671,
            "laggy_interval": 18,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 23,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.849840,
            "laggy_interval": 65,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 24,
            "down_stamp": "2015-04-26 02:03:26.390577",
            "laggy_probability": 0.806616,
            "laggy_interval": 47,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 25,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.810979,
            "laggy_interval": 52,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 26,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.858681,
            "laggy_interval": 132,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 27,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.796432,
            "laggy_interval": 70,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 28,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.908335,
            "laggy_interval": 39,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 29,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.732148,
            "laggy_interval": 53,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 30,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.866609,
            "laggy_interval": 44,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 31,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.866052,
            "laggy_interval": 46,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 32,
            "down_stamp": "2015-04-26 02:21:21.502025",
            "laggy_probability": 0.812573,
            "laggy_interval": 53,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 33,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.797734,
            "laggy_interval": 62,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 34,
            "down_stamp": "2015-04-26 14:21:27.940755",
            "laggy_probability": 0.681026,
            "laggy_interval": 108,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 35,
            "down_stamp": "2015-04-26 02:03:26.390577",
            "laggy_probability": 0.703038,
            "laggy_interval": 47,
            "features": 1125899906842623,
            "old_weight": 65536
        },
        {
            "osd": 36,
            "down_stamp": "2015-04-27 16:35:38.560828",
            "laggy_probability": 0.286094,
            "laggy_interval": 6,
            "features": 1125899906842623,
            "old_weight": 0
        }
    ],
    "pg_temp": [],
    "primary_temp": [],
    "blacklist": [],
    "erasure_code_profiles": {
        "default": {
            "directory": "\/usr\/lib\/ceph\/erasure-code",
            "k": "2",
            "m": "1",
            "plugin": "jerasure",
            "technique": "reed_sol_van"
        }
    }
}
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux