Thank you for the explanation, Brad. I will change that setting and see how it goes.
Subhachandra
On Thu, Jan 11, 2018 at 10:38 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote:
On Fri, Jan 12, 2018 at 11:27 AM, Subhachandra Chandra
<schandra@xxxxxxxxxxxx> wrote:
> Hello,
>
> We are running experiments on a Ceph cluster before we move data on it.
> While trying to increase the number of PGs on one of the pools it threw the
> following error
>
> root@ctrl1:/# ceph osd pool set data pg_num 65536
> Error E2BIG: specified pg_num 65536 is too large (creating 32768 new PGs on
> ~540 OSDs exceeds per-OSD max of 32)
That comes from here:
https://github.com/ceph/ceph/blob/ 5d7813f612aea59239c8375aaa0091 9ae32f952f/src/mon/OSDMonitor. cc#L6027
So the warning is triggered because new_pgs (65536) >
g_conf->mon_osd_max_split_count (32) * expected_osds (540)
Maybe the wording could be better. Perhaps "exceeds per-OSD max with
>
> There are 2 pools named "data" and "metadata". "data" is an erasure coded
> pool (6,3) and "metadata" is a replicated pool with a replication factor of
> 3.
>
> root@ctrl1:/# ceph osd lspools
> 1 metadata,2 data,
> root@ctrl1:/# ceph osd pool get metadata pg_num
> pg_num: 512
> root@ctrl1:/# ceph osd pool get data pg_num
> pg_num: 32768
>
> osd: 540 osds: 540 up, 540 in
> flags noout,noscrub,nodeep-scrub
>
> data:
> pools: 2 pools, 33280 pgs
> objects: 7090k objects, 1662 TB
> usage: 2501 TB used, 1428 TB / 3929 TB avail
> pgs: 33280 active+clean
>
> The current PG/OSD ratio according to my calculation should be 549
>>>> (32768 * 9 + 512 * 3 ) / 540.0
> 548.9777777777778
>
> Increasing the number of PGs in the "data" pool should increase the PG/OSD
> ratio to about 1095
>>>> (65536 * 9 + 512 * 3 ) / 540.0
> 1095.111111111111
>
> In the config, settings related to PG/OSD ratio look like
> mon_max_pg_per_osd = 1500
> osd_max_pg_per_osd_hard_ratio = 1.0
>
> Trying to increase the number of PGs to 65536 throws the previously
> mentioned error. The new PG/OSD ratio is still under the configured limit.
> Why do we see the error? Further, there seems to be a bug in the error
> message where it says "exceeds per-OSD max of 32" in terms of where does
> "32" comes from?
mon_osd_max_split_count of 32". I'll submit this and see how it goes.
>
> P.S. I understand that the PG/OSD ratio configured on this cluster far
> exceeds the recommended values. The experiment is to find scaling limits and
> try out expansion scenarios.
>
> Thanks
> Subhachandra
>
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>
--
Cheers,
Brad
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
