Hi,
Mabye this info can help to find what is wrong.
For one PG (3.1e4a) which is active+remapped:
{ "state": "active+remapped",
"epoch": 96050,
"up": [
119,
69],
"acting": [
119,
69,
7],
Logs:
On osd.7:
2014-02-04 09:45:54.966913 7fa618afe700 1 osd.7 pg_epoch: 94460
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=93486
n=6718 ec=4 les/c 93486/93486 94460/94460/92233) [119,69] r=-1
lpr=94460 pi=92546-94459/5 lcod 94459'207003 inactive NOTIFY]
state<Start>: transitioning to Stray
2014-02-04 09:45:55.781278 7fa6172fb700 1 osd.7 pg_epoch: 94461
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=93486
n=6718 ec=4 les/c 93486/93486 94460/94461/92233)
[119,69]/[119,69,7,142] r=2 lpr=94461 pi=92546-94460/6 lcod
94459'207003 remapped NOTIFY] state<Start>: transitioning to Stray
2014-02-04 09:49:01.124510 7fa618afe700 1 osd.7 pg_epoch: 94495
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=94462
n=6718 ec=4 les/c 94462/94494 94460/94495/92233) [119,69]/[119,69,7]
r=2 lpr=94495 pi=92546-94494/7 lcod 94459'207003 remapped]
state<Start>: transitioning to Stray
On osd.119:
2014-02-04 09:45:54.981707 7f37f07c5700 1 osd.119 pg_epoch: 94460
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=93486
n=6718 ec=4 les/c 93486/93486 94460/94460/92233) [119,69] r=0
lpr=94460 pi=93485-94459/1 mlcod 0'0 inactive] state<Start>:
transitioning to Primary
2014-02-04 09:45:55.805712 7f37ecfbe700 1 osd.119 pg_epoch: 94461
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=93486
n=6718 ec=4 les/c 93486/93486 94460/94461/92233)
[119,69]/[119,69,7,142] r=0 lpr=94461 pi=93485-94460/2 mlcod 0'0
remapped] state<Start>: transitioning to Primary
2014-02-04 09:45:56.794015 7f37edfc0700 0 log [INF] : 3.1e4a
restarting backfill on osd.69 from (0'0,0'0] MAX to 94459'207004
2014-02-04 09:49:01.156627 7f37ef7c3700 1 osd.119 pg_epoch: 94495
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=94462
n=6718 ec=4 les/c 94462/94494 94460/94495/92233) [119,69]/[119,69,7]
r=0 lpr=94495 pi=94461-94494/1 mlcod 0'0 remapped] state<Start>:
transitioning to Primary
On osd.69:
2014-02-04 09:45:56.845695 7f2231372700 1 osd.69 pg_epoch: 94462
pg[3.1e4a( empty local-les=0 n=0 ec=4 les/c 93486/93486
94460/94461/92233) [119,69]/[119,69,7,142] r=1 lpr=94462
pi=93485-94460/2 inactive] state<Start>: transitioning to Stray
2014-02-04 09:49:01.153695 7f2229b63700 1 osd.69 pg_epoch: 94495
pg[3.1e4a( v 94459'207004 (72275'204004,94459'207004] local-les=94462
n=6718 ec=4 les/c 94462/94494 94460/94495/92233) [119,69]/[119,69,7]
r=1 lpr=94495 pi=93485-94494/3 remapped] state<Start>: transitioning
to Stray
pq query recovery state:
"recovery_state": [
{ "name": "Started\/Primary\/Active",
"enter_time": "2014-02-04 09:49:02.070724",
"might_have_unfound": [],
"recovery_progress": { "backfill_target": -1,
"waiting_on_backfill": 0,
"backfill_pos": "0\/\/0\/\/-1",
"backfill_info": { "begin": "0\/\/0\/\/-1",
"end": "0\/\/0\/\/-1",
"objects": []},
"peer_backfill_info": { "begin": "0\/\/0\/\/-1",
"end": "0\/\/0\/\/-1",
"objects": []},
"backfills_in_flight": [],
"pull_from_peer": [],
"pushing": []},
"scrub": { "scrubber.epoch_start": "77502",
"scrubber.active": 0,
"scrubber.block_writes": 0,
"scrubber.finalizing": 0,
"scrubber.waiting_on": 0,
"scrubber.waiting_on_whom": []}},
{ "name": "Started",
"enter_time": "2014-02-04 09:49:01.156626"}]}
---
Regards
Dominik
2014-02-04 12:09 GMT+01:00 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>:
> Hi,
> Thanks for Your help !!
> We've done again 'ceph osd reweight-by-utilization 105'
> Cluster stack on 10387 active+clean, 237 active+remapped;
> More info in attachments.
>
> --
> Regards
> Dominik
>
>
> 2014-02-04 Sage Weil <sage@xxxxxxxxxxx>:
>> Hi,
>>
>> I spent a couple hours looking at your map because it did look like there
>> was something wrong. After some experimentation and adding a bucnh of
>> improvements to osdmaptool to test the distribution, though, I think
>> everything is working as expected. For pool 3, your map has a standard
>> deviation in utilizations of ~8%, and we should expect ~9% for this number
>> of PGs. For all pools, it is slightly higher (~9% vs expected ~8%).
>> This is either just in the noise, or slightly confounded by the lack of
>> the hashpspool flag on the pools (which slightly amplifies placement
>> nonuniformity with multiple pools... not enough that it is worth changing
>> anything though).
>>
>> The bad news is that that order of standard deviation results in pretty
>> wide min/max range of 118 to 202 pgs. That seems a *bit* higher than we a
>> perfectly random placement generates (I'm seeing a spread in that is
>> usually 50-70 pgs), but I think *that* is where the pool overlap (no
>> hashpspool) is rearing its head; for just pool three the spread of 50 is
>> about what is expected.
>>
>> Long story short: you have two options. One is increasing the number of
>> PGs. Note that this helps but has diminishing returns (doubling PGs
>> only takes you from ~8% to ~6% standard deviation, quadrupling to ~4%).
>>
>> The other is to use reweight-by-utilization. That is the best approach,
>> IMO. I'm not sure why you were seeing PGs stuck in the remapped state
>> after you did that, though, but I'm happy to dig into that too.
>>
>> BTW, the osdmaptool addition I was using to play with is here:
>> https://github.com/ceph/ceph/pull/1178
>>
>> sage
>>
>>
>> On Mon, 3 Feb 2014, Dominik Mostowiec wrote:
>>
>>> In other words,
>>> 1. we've got 3 racks ( 1 replica per rack )
>>> 2. in every rack we have 3 hosts
>>> 3. every host has 22 OSD's
>>> 4. all pg_num's are 2^n for every pool
>>> 5. we enabled "crush tunables optimal".
>>> 6. on every machine we disabled 4 unused disk's (osd out, osd reweight
>>> 0 and osd rm)
>>>
>>> Pool ".rgw.buckets": one osd has 105 PGs and other one (on the same
>>> machine) has 144 PGs (37% more!).
>>> Other pools also have got this problem. It's not efficient placement.
>>>
>>> --
>>> Regards
>>> Dominik
>>>
>>>
>>> 2014-02-02 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>:
>>> > Hi,
>>> > For more info:
>>> > crush: http://dysk.onet.pl/link/r4wGK
>>> > osd_dump: http://dysk.onet.pl/link/I3YMZ
>>> > pg_dump: http://dysk.onet.pl/link/4jkqM
>>> >
>>> > --
>>> > Regards
>>> > Dominik
>>> >
>>> > 2014-02-02 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>:
>>> >> Hi,
>>> >> Hmm,
>>> >> You think about sumarize PGs from different pools on one OSD's i think.
>>> >> But for one pool (.rgw.buckets) where i have almost of all my data, PG
>>> >> count on OSDs is aslo different.
>>> >> For example 105 vs 144 PGs from pool .rgw.buckets. In first case it is
>>> >> 52% disk usage, second 74%.
>>> >>
>>> >> --
>>> >> Regards
>>> >> Dominik
>>> >>
>>> >>
>>> >> 2014-02-02 Sage Weil <sage@xxxxxxxxxxx>:
>>> >>> It occurs to me that this (and other unexplain variance reports) could
>>> >>> easily be the 'hashpspool' flag not being set. The old behavior had the
>>> >>> misfeature where consecutive pool's pg's would 'line up' on the same osds,
>>> >>> so that 1.7 == 2.6 == 3.5 == 4.4 etc would map to the same nodes. This
>>> >>> tends to 'amplify' any variance in the placement. The default is still to
>>> >>> use the old behavior for compatibility (this will finally change in
>>> >>> firefly).
>>> >>>
>>> >>> You can do
>>> >>>
>>> >>> ceph osd pool set <poolname> hashpspool true
>>> >>>
>>> >>> to enable the new placement logic on an existing pool, but be warned that
>>> >>> this will rebalance *all* of the data in the pool, which can be a very
>>> >>> heavyweight operation...
>>> >>>
>>> >>> sage
>>> >>>
>>> >>>
>>> >>> On Sun, 2 Feb 2014, Dominik Mostowiec wrote:
>>> >>>
>>> >>>> Hi,
>>> >>>> After scrubbing almost all PGs has equal(~) num of objects.
>>> >>>> I found something else.
>>> >>>> On one host PG coun on OSDs:
>>> >>>> OSD with small(52%) disk usage:
>>> >>>> count, pool
>>> >>>> 105 3
>>> >>>> 18 4
>>> >>>> 3 5
>>> >>>>
>>> >>>> Osd with larger(74%) disk usage:
>>> >>>> 144 3
>>> >>>> 31 4
>>> >>>> 2 5
>>> >>>>
>>> >>>> Pool 3 is .rgw.buckets (where is almost of all data).
>>> >>>> Pool 4 is .log, where is no data.
>>> >>>>
>>> >>>> Count of PGs shouldn't be the same per OSD ?
>>> >>>> Or maybe PG hash algorithm is disrupted by wrong count of PG for pool
>>> >>>> '4'. There is 1440 PGs ( this is not power of 2 ).
>>> >>>>
>>> >>>> ceph osd dump:
>>> >>>> pool 0 'data' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 64 pgp_num 64 last_change 28459 owner 0
>>> >>>> crash_replay_interval 45
>>> >>>> pool 1 'metadata' rep size 3 min_size 1 crush_ruleset 1 object_hash
>>> >>>> rjenkins pg_num 64 pgp_num 64 last_change 28460 owner 0
>>> >>>> pool 2 'rbd' rep size 3 min_size 1 crush_ruleset 2 object_hash
>>> >>>> rjenkins pg_num 64 pgp_num 64 last_change 28461 owner 0
>>> >>>> pool 3 '.rgw.buckets' rep size 3 min_size 1 crush_ruleset 0
>>> >>>> object_hash rjenkins pg_num 8192 pgp_num 8192 last_change 73711 owner
>>> >>>> 0
>>> >>>> pool 4 '.log' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 1440 pgp_num 1440 last_change 28463 owner 0
>>> >>>> pool 5 '.rgw' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 128 pgp_num 128 last_change 72467 owner 0
>>> >>>> pool 6 '.users.uid' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 8 pgp_num 8 last_change 28465 owner 0
>>> >>>> pool 7 '.users' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 8 pgp_num 8 last_change 28466 owner 0
>>> >>>> pool 8 '.usage' rep size 2 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 8 pgp_num 8 last_change 28467 owner
>>> >>>> 18446744073709551615
>>> >>>> pool 9 '.intent-log' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 8 pgp_num 8 last_change 28468 owner
>>> >>>> 18446744073709551615
>>> >>>> pool 10 '.rgw.control' rep size 3 min_size 1 crush_ruleset 0
>>> >>>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 33485 owner
>>> >>>> 18446744073709551615
>>> >>>> pool 11 '.rgw.gc' rep size 3 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 8 pgp_num 8 last_change 33487 owner
>>> >>>> 18446744073709551615
>>> >>>> pool 12 '.rgw.root' rep size 2 min_size 1 crush_ruleset 0 object_hash
>>> >>>> rjenkins pg_num 8 pgp_num 8 last_change 44540 owner 0
>>> >>>> pool 13 '' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
>>> >>>> pg_num 8 pgp_num 8 last_change 46912 owner 0
>>> >>>>
>>> >>>> --
>>> >>>> Regards
>>> >>>> Dominik
>>> >>>>
>>> >>>> 2014-02-01 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>:
>>> >>>> > Hi,
>>> >>>> >> Did you bump pgp_num as well?
>>> >>>> > Yes.
>>> >>>> >
>>> >>>> > See: http://dysk.onet.pl/link/BZ968
>>> >>>> >
>>> >>>> >> 25% pools is two times smaller from other.
>>> >>>> > This is changing after scrubbing.
>>> >>>> >
>>> >>>> > --
>>> >>>> > Regards
>>> >>>> > Dominik
>>> >>>> >
>>> >>>> > 2014-02-01 Kyle Bader <kyle.bader@xxxxxxxxx>:
>>> >>>> >>
>>> >>>> >>> Change pg_num for .rgw.buckets to power of 2, an 'crush tunables
>>> >>>> >>> optimal' didn't help :(
>>> >>>> >>
>>> >>>> >> Did you bump pgp_num as well? The split pgs will stay in place until pgp_num
>>> >>>> >> is bumped as well, if you do this be prepared for (potentially lots) of data
>>> >>>> >> movement.
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > --
>>> >>>> > Pozdrawiam
>>> >>>> > Dominik
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Pozdrawiam
>>> >>>> Dominik
>>> >>>>
>>> >>>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Pozdrawiam
>>> >> Dominik
>>> >
>>> >
>>> >
>>> > --
>>> > Pozdrawiam
>>> > Dominik
>>>
>>>
>>>
>>> --
>>> Pozdrawiam
>>> Dominik
>>>
>>>
>
>
>
> --
> Pozdrawiam
> Dominik
--
Pozdrawiam
Dominik
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
