Dilemma with PG distribution

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,
I am just evaluating out cluster configuration again, because we had an
very bad incident with laggy OSDs that shut down the entire cluster.

We use datacenter SSDs in different sizes (2, 4, 8TB) and someone said,
that I should not go beyond a specific amount of PGs on certain device
classes.

Now the distribution in our cluster is 100PGs on the 2TB disks, 200PGs on
the 4TB disks and 400PGs on the 8TB disks.

According to this (
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#choosing-the-number-of-placement-groups)
I should got with 150OSDs * 100 / 3 replication and round UP to the nearest
power of two which would be 8k

BUT because these are just SSDs (100k/30k read/write IOPS) I think I
should  got down to 4k PGs, to not overload the 8TB disks with operations.

What do you think?

These are out cluster stats:
# ceph df
--- RAW STORAGE ---
CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
ssd    419 TiB  122 TiB  297 TiB   298 TiB      71.01
TOTAL  419 TiB  122 TiB  297 TiB   298 TiB      71.01

--- POOLS ---
POOL                   ID  PGS   STORED   OBJECTS  USED     %USED  MAX AVAIL
isos                    7    64  431 GiB  111.58k  1.3 TiB   1.57     26 TiB
rbd                     8  8192   98 TiB   34.17M  288 TiB  78.46     26 TiB
archive                 9   128  2.6 TiB  722.10k  7.8 TiB   8.99     26 TiB
device_health_metrics  10     1  203 MiB      156  608 MiB      0     26 TiB

# ceph status
  cluster:
    id:     74313356-3b3d-43f3-bce6-9fb0e4591097
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-rbd-mon4,ceph-rbd-mon5,ceph-rbd-mon6 (age
44h)
    mgr: ceph-rbd-mon5(active, since 44h), standbys: ceph-rbd-mon6,
ceph-rbd-mon4
    osd: 150 osds: 150 up (since 44h), 150 in (since 6d)

  data:
    pools:   4 pools, 8385 pgs
    objects: 35.01M objects, 110 TiB
    usage:   298 TiB used, 122 TiB / 419 TiB avail
    pgs:     8380 active+clean
             3    active+clean+scrubbing+deep
             2    active+clean+scrubbing

  io:
    client:   100 MiB/s rd, 433 MiB/s wr, 1.45k op/s rd, 12.02k op/s wr

-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux