Re: Degraded data redundancy and too many PGs per OSD

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

how did you end up with that many PGs per OSD? According to your output the pg_autoscaler is enabled, if that was done by the autoscaler I would create a tracker issue for that. Then I would either disable it or set the mode to "warn" and then reduce the pg_num for some of the pools. What does your crush rule 2 look like? Can you share the dump of the rule with the ID 2?

ceph osd crush rule ls
ceph osd crush rule dump <NAME>


Zitat von farhad kh <farhad.khedriyan@xxxxxxxxx>:

hi
i have a problem in my cluster
i used cache tier for rgw data
In this way, three hosts for cache and three hosts for data I have used
SSDs for cache and HDD for data
i set 20 GiB quota for cache pool
when one host of cache tier shulde be offline
released this warning and i decreased quota to 10 GiB but it not resolved
and in dashboard not correct number of pg status ( 1 active+undersize)
what happening in my cluster ?
why this is not resolved?
anyone can explain this situation?
------------------------------------------------
##ceph -s
opcpmfpsksa0101: Mon May 30 12:05:12 2022

  cluster:
    id:     54d2b1d6-207e-11ec-8c73-005056ac51bf
    health: HEALTH_WARN
            1 hosts fail cephadm check
            1 pools have many more objects per pg than average
            Degraded data redundancy: 1750/53232 objects degraded (3.287%),
1 pg degraded, 1 pg undersized
            too many PGs per OSD (259 > max 250)

  services:
    mon: 3 daemons, quorum opcpmfpsksa0101,opcpmfpsksa0103,opcpmfpsksa0105
(age 3d)
    mgr: opcpmfpsksa0101.apmwdm(active, since 5h)
    osd: 12 osds: 10 up (since 95m), 10 in (since 85m)
    rgw: 2 daemons active (2 hosts, 1 zones)

  data:
    pools:   9 pools, 865 pgs
    objects: 17.74k objects, 41 GiB
    usage:   128 GiB used, 212 GiB / 340 GiB avail
    pgs:     1750/53232 objects degraded (3.287%)
             864 active+clean
             1   active+undersized+degraded

-----------------------------
## ceph health detail
HEALTH_WARN 1 hosts fail cephadm check; 1 pools have many more objects per
pg than average; Degraded data redundancy: 1665/56910 objects degraded
(2.926%), 1 pg degraded, 1 pg undersized; too many PGs per OSD (259 > max
250)
[WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check
    host opcpcfpsksa0101 (10.56.12.210) failed check: Failed to connect to
opcpcfpsksa0101 (10.56.12.210).
Please make sure that the host is reachable and accepts connections using
the cephadm SSH key

To add the cephadm SSH key to the host:
ceph cephadm get-pub-key > ~/ceph.pub
ssh-copy-id -f -i ~/ceph.pub root@10.56.12.210

To check that the host is reachable open a new shell with the --no-hosts
flag:
cephadm shell --no-hosts

Then run the following:
ceph cephadm get-ssh-config > ssh_config
ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
chmod 0600 ~/cephadm_private_key
ssh -F ssh_config -i ~/cephadm_private_key root@10.56.12.210
[WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than
average
    pool cache-pool objects per pg (1665) is more than 79.2857 times
cluster average (21)
[WRN] PG_DEGRADED: Degraded data redundancy: 1665/56910 objects degraded
(2.926%), 1 pg degraded, 1 pg undersized
    pg 9.0 is stuck undersized for 88m, current state
active+undersized+degraded, last acting [10,11]
[WRN] TOO_MANY_PGS: too many PGs per OSD (259 > max 250)
--------------------------------------------------
ceph osd df tree
ID   CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META
AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
 -1         0.35156         -  340 GiB  128 GiB  121 GiB   12 MiB  6.9 GiB
 212 GiB  37.58  1.00    -          root default
 -3         0.01959         -      0 B      0 B      0 B      0 B      0 B
     0 B      0     0    -              host opcpcfpsksa0101
  0    ssd  0.00980         0      0 B      0 B      0 B      0 B      0 B
     0 B      0     0    0    down          osd.0
  9    ssd  0.00980         0      0 B      0 B      0 B      0 B      0 B
     0 B      0     0    0    down          osd.9
 -5         0.01959         -   20 GiB  5.1 GiB  4.0 GiB  588 KiB  1.1 GiB
  15 GiB  25.29  0.67    -              host opcpcfpsksa0103
  7    ssd  0.00980   0.85004   10 GiB  483 MiB   75 MiB  539 KiB  407 MiB
 9.5 GiB   4.72  0.13    3      up          osd.7
 10    ssd  0.00980   0.55011   10 GiB  4.6 GiB  3.9 GiB   49 KiB  703 MiB
 5.4 GiB  45.85  1.22    5      up          osd.10
-16         0.01959         -   20 GiB  5.5 GiB  4.0 GiB  542 KiB  1.5 GiB
  15 GiB  27.28  0.73    -              host opcpcfpsksa0105
  8    ssd  0.00980   0.70007   10 GiB  851 MiB   75 MiB  121 KiB  775 MiB
 9.2 GiB   8.31  0.22   10      up          osd.8
 11    ssd  0.00980   0.45013   10 GiB  4.6 GiB  3.9 GiB  421 KiB  742 MiB
 5.4 GiB  46.24  1.23    5      up          osd.11
-10         0.09760         -  100 GiB   39 GiB   38 GiB  207 KiB  963 MiB
  61 GiB  38.59  1.03    -              host opcsdfpsksa0101
  1    hdd  0.04880   1.00000   50 GiB   19 GiB   19 GiB  207 KiB  639 MiB
  31 GiB  38.77  1.03  424      up          osd.1
 12    hdd  0.04880   1.00000   50 GiB   19 GiB   19 GiB      0 B  323 MiB
  31 GiB  38.40  1.02  430      up          osd.12
-13         0.09760         -  100 GiB   39 GiB   38 GiB  4.9 MiB  1.8 GiB
  61 GiB  39.41  1.05    -              host opcsdfpsksa0103
  2    hdd  0.04880   1.00000   50 GiB   20 GiB   20 GiB  2.6 MiB  703 MiB
  30 GiB  40.42  1.08  428      up          osd.2
  3    hdd  0.04880   1.00000   50 GiB   19 GiB   18 GiB  2.3 MiB  1.1 GiB
  31 GiB  38.39  1.02  429      up          osd.3
-19         0.09760         -  100 GiB   39 GiB   38 GiB  5.3 MiB  1.6 GiB
  61 GiB  39.25  1.04    -              host opcsdfpsksa0105
  4    hdd  0.04880   1.00000   50 GiB   19 GiB   19 GiB  2.7 MiB  560 MiB
  31 GiB  38.21  1.02  433      up          osd.4
  5    hdd  0.04880   1.00000   50 GiB   20 GiB   19 GiB  2.6 MiB  1.1 GiB
  30 GiB  40.30  1.07  427      up          osd.5
                        TOTAL  340 GiB  128 GiB  121 GiB   12 MiB  6.9 GiB
 212 GiB  37.58
MIN/MAX VAR: 0.13/1.23  STDDEV: 13.71
--------------------------------

 rados df
POOL_NAME                      USED  OBJECTS  CLONES  COPIES
 MISSING_ON_PRIMARY  UNFOUND  DEGRADED    RD_OPS       RD    WR_OPS
WR  USED COMPR  UNDER COMPR
.rgw.root                    52 KiB        4       0      12
    0        0         0       724  728 KiB        48   27 KiB         0 B
         0 B
cache-pool                  8.6 GiB     1750       0    5250
    0        0      1750       449  265 MiB      4003   20 KiB         0 B
         0 B
default.rgw.buckets.data    112 GiB    15498       0   46494
    0        0         0    249164   48 GiB   1241400  545 GiB         0 B
         0 B
default.rgw.buckets.index   6.6 MiB       11       0      33
    0        0         0     82342  245 MiB     33831   17 MiB         0 B
         0 B
default.rgw.buckets.non-ec  7.5 MiB      250       0     750
    0        0         0    524383  320 MiB    149486  124 MiB         0 B
         0 B
default.rgw.control             0 B        8       0      24
    0        0         0         0      0 B         0      0 B         0 B
         0 B
default.rgw.log             1.2 MiB      209       0     627
    0        0         0  26272099   25 GiB  17227290   69 MiB         0 B
         0 B
default.rgw.meta            121 KiB       14       0      42
    0        0         0      3007  2.2 MiB       129   58 KiB         0 B
         0 B
device_health_metrics           0 B        0       0       0
    0        0         0         0      0 B         0      0 B         0 B
         0 B

total_objects    17744
total_used       128 GiB
total_avail      212 GiB
total_space      340 GiB
---------------------------------------------------
pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change
17591 lfor 0/14881/15285 flags hashpspool stripe_width 0 pg_num_min 1
application mgr_devicehealth
pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 1 object_hash
rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 17835 lfor
0/3320/3318 flags hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
17840 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
17847 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change
17855 lfor 0/15243/15287 flags hashpspool stripe_width 0 pg_autoscale_bias
4 pg_num_min 8 application rgw
pool 6 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
1 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change
17590 lfor 0/15226/15287 flags hashpspool stripe_width 0 pg_autoscale_bias
4 pg_num_min 8 application rgw
pool 7 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 1
object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change
17826 lfor 17826/17826/17826 flags hashpspool tiers 9 read_tier 9
write_tier 9 stripe_width 0 application rgw
pool 8 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule
1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change
17555 flags hashpspool stripe_width 0 application rgw
pool 9 'cache-pool' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 18019 lfor
17826/17826/17826 flags hashpspool,incomplete_clones max_bytes 10737418240
tier_of 7 cache_mode writeback target_bytes 7516192768 hit_set
bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 300s x8
decay_rate 0 search_last_n 0 stripe_width 0 application rgw
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux