+ceph-users On Mon, 6 Jan 2025 at 15:50, Bruno Gomes Pessanha <bruno.pessanha@xxxxxxxxx> wrote: > `ceph osd df`, look at the PGS column. Could be you’re hitting the limit >> on some OSDs. It’s odd to stop at a non power of 2. >> > > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS > 8 ssd 13.97069 1.00000 14 TiB 9.4 TiB 9.4 TiB 9.8 MiB 21 > GiB 4.6 TiB 67.09 1.07 63 up > 18 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 8.5 MiB 19 > GiB 5.7 TiB 59.02 0.94 54 up > 28 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 1.5 GiB 19 > GiB 5.9 TiB 58.02 0.93 54 up > 37 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 84 MiB 31 > GiB 1.2 TiB 91.14 1.45 135 up > 45 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 542 MiB 22 > GiB 4.8 TiB 65.79 1.05 66 up > 56 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 5.0 MiB 19 > GiB 5.7 TiB 59.19 0.94 56 up > 66 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.8 TiB 358 MiB 20 > GiB 5.1 TiB 63.39 1.01 54 up > 77 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 3.7 MiB 19 > GiB 5.3 TiB 62.18 0.99 53 up > 6 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 355 MiB 18 > GiB 5.6 TiB 59.67 0.95 52 up > 16 ssd 13.97069 1.00000 14 TiB 7.3 TiB 7.3 TiB 572 MiB 18 > GiB 6.7 TiB 52.05 0.83 47 up > 26 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 921 MiB 20 > GiB 5.4 TiB 61.55 0.98 57 up > 36 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.2 GiB 31 > GiB 1.2 TiB 91.45 1.46 141 up > 46 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 419 MiB 19 > GiB 5.9 TiB 57.77 0.92 57 up > 54 ssd 13.97069 1.00000 14 TiB 9.0 TiB 8.9 TiB 1011 MiB 20 > GiB 5.0 TiB 64.10 1.02 65 up > 64 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 962 MiB 19 > GiB 6.0 TiB 56.87 0.91 59 up > 74 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 879 MiB 17 > GiB 6.8 TiB 51.23 0.82 49 up > 3 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.1 TiB 2.0 GiB 20 > GiB 4.8 TiB 65.45 1.04 71 up > 12 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 433 MiB 19 > GiB 5.6 TiB 59.92 0.96 61 up > 22 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.8 TiB 501 MiB 20 > GiB 5.2 TiB 63.01 1.01 66 up > 31 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 932 MiB 18 > GiB 5.8 TiB 58.20 0.93 53 up > 41 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.6 TiB 627 MiB 20 > GiB 5.3 TiB 61.96 0.99 54 up > 51 ssd 13.97069 1.00000 14 TiB 6.9 TiB 6.9 TiB 1.4 GiB 16 > GiB 7.1 TiB 49.18 0.78 43 up > 61 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1000 MiB 31 > GiB 1.4 TiB 89.99 1.44 134 up > 71 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 553 MiB 19 > GiB 5.9 TiB 57.63 0.92 51 up > 4 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 525 MiB 15 > GiB 6.9 TiB 50.87 0.81 47 up > 15 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 606 MiB 19 > GiB 5.2 TiB 62.55 1.00 55 up > 23 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.8 TiB 423 MiB 18 > GiB 6.1 TiB 56.32 0.90 49 up > 34 ssd 13.97069 1.00000 14 TiB 10 TiB 10 TiB 551 MiB 24 > GiB 3.5 TiB 74.79 1.19 72 up > 44 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 77 MiB 28 > GiB 1.4 TiB 89.90 1.43 132 up > 59 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.0 TiB 510 MiB 20 > GiB 4.9 TiB 64.91 1.04 61 up > 69 ssd 13.97069 1.00000 14 TiB 9.7 TiB 9.7 TiB 1.0 GiB 22 > GiB 4.2 TiB 69.62 1.11 71 up > 79 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 1.0 GiB 17 > GiB 6.8 TiB 51.26 0.82 47 up > 9 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 1.1 MiB 18 > GiB 5.8 TiB 58.81 0.94 49 up > 19 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 421 MiB 21 > GiB 4.8 TiB 65.68 1.05 63 up > 29 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.7 TiB 511 MiB 21 > GiB 5.2 TiB 62.68 1.00 61 up > 39 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 556 MiB 20 > GiB 5.6 TiB 60.15 0.96 58 up > 49 ssd 13.97069 1.00000 14 TiB 7.0 TiB 7.0 TiB 2.9 MiB 15 > GiB 7.0 TiB 49.92 0.80 45 up > 58 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 432 MiB 16 > GiB 6.8 TiB 51.22 0.82 49 up > 67 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.1 GiB 17 > GiB 6.6 TiB 52.80 0.84 47 up > 78 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 574 MiB 19 > GiB 6.3 TiB 54.94 0.88 55 up > 0 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 455 MiB 19 > GiB 5.8 TiB 58.66 0.94 58 up > 10 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 439 MiB 19 > GiB 5.4 TiB 61.07 0.97 48 up > 21 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 646 MiB 30 > GiB 1.3 TiB 90.87 1.45 135 up > 32 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 3.2 MiB 19 > GiB 5.4 TiB 61.60 0.98 56 up > 42 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 582 MiB 33 > GiB 1.3 TiB 90.78 1.45 133 up > 52 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.4 GiB 17 > GiB 6.6 TiB 52.74 0.84 47 up > 62 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 1.5 GiB 16 > GiB 6.6 TiB 52.68 0.84 49 up > 72 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 1.2 GiB 17 > GiB 6.3 TiB 55.20 0.88 53 up > 1 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 1.5 GiB 19 > GiB 5.9 TiB 57.73 0.92 64 up > 11 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 422 MiB 19 > GiB 5.5 TiB 60.37 0.96 62 up > 20 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 405 MiB 17 > GiB 6.3 TiB 55.02 0.88 52 up > 30 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 494 MiB 19 > GiB 5.9 TiB 57.84 0.92 49 up > 40 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 376 MiB 22 > GiB 4.7 TiB 66.17 1.06 74 up > 50 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.0 GiB 20 > GiB 5.7 TiB 59.43 0.95 54 up > 60 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 2.1 GiB 32 > GiB 1.2 TiB 91.27 1.46 136 up > 70 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 498 MiB 19 > GiB 5.6 TiB 59.64 0.95 56 up > 2 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.1 GiB 31 > GiB 1.2 TiB 91.62 1.46 133 up > 14 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 9.8 MiB 22 > GiB 4.7 TiB 66.05 1.05 60 up > 24 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 426 MiB 17 > GiB 6.5 TiB 53.13 0.85 50 up > 33 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.9 TiB 413 MiB 20 > GiB 5.1 TiB 63.54 1.01 56 up > 43 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 529 MiB 18 > GiB 5.8 TiB 58.58 0.93 58 up > 53 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.6 TiB 533 MiB 17 > GiB 6.4 TiB 54.29 0.87 52 up > 63 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 569 MiB 19 > GiB 5.4 TiB 61.20 0.98 58 up > 73 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 536 MiB 20 > GiB 5.3 TiB 62.14 0.99 63 up > 5 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.3 GiB 19 > GiB 5.7 TiB 59.46 0.95 53 up > 13 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 923 MiB 21 > GiB 5.4 TiB 61.60 0.98 49 up > 25 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 4.3 MiB 16 > GiB 6.6 TiB 53.03 0.85 46 up > 35 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 397 MiB 17 > GiB 6.6 TiB 52.70 0.84 45 up > 47 ssd 13.97069 1.00000 14 TiB 9.0 TiB 9.0 TiB 10 MiB 21 > GiB 4.9 TiB 64.74 1.03 60 up > 57 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 418 MiB 17 > GiB 6.8 TiB 51.13 0.82 50 up > 68 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 590 MiB 30 > GiB 1.2 TiB 91.28 1.46 141 up > 76 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 976 MiB 16 > GiB 6.7 TiB 51.70 0.82 46 up > 7 ssd 13.97069 1.00000 14 TiB 7.8 TiB 7.8 TiB 2.6 MiB 18 > GiB 6.1 TiB 56.03 0.89 49 up > 17 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 443 MiB 16 > GiB 6.8 TiB 51.34 0.82 43 up > 27 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.4 TiB 8.6 MiB 21 > GiB 5.5 TiB 60.51 0.97 54 up > 38 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 1005 MiB 18 > GiB 6.0 TiB 56.88 0.91 52 up > 48 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.5 TiB 2.0 MiB 17 > GiB 6.4 TiB 54.08 0.86 45 up > 55 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 509 MiB 30 > GiB 1.2 TiB 91.50 1.46 137 up > 65 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 901 MiB 17 > GiB 6.3 TiB 55.07 0.88 51 up > 75 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 6.0 MiB 19 > GiB 5.5 TiB 60.66 0.97 58 up > TOTAL 1.1 PiB 701 TiB 699 TiB 49 GiB 1.6 > TiB 417 TiB 62.70 > MIN/MAX VAR: 0.78/1.46 STDDEV: 11.76 > > We’re a community! That curve is way gentler than it used to be. >> > 🙌👍 > > > On Sun, 5 Jan 2025 at 14:59, Anthony D'Atri <anthony.datri@xxxxxxxxx> > wrote: > >> >> >> Do you use the autoscalar or did you trigger a manual PG increment of >> the >> >> pool? >> > >> > The pool had autoscale enabled until 2 days ago when I thought it was >> > better to change things manually in order to have a more deterministic >> > result. Yes, I wanted to increase from "1" to something like "1024" but >> it >> > looks like it was capped to the 144 no matter what I do: >> >> `ceph osd df`, look at the PGS column. Could be you’re hitting the limit >> on some OSDs. It’s odd to stop at a non power of 2. >> >> > Is it correct to say that every PG/OSD change can potentially cause data >> > misplacements, unbalanced osd's and long backfills? I'll be way more >> > careful before tuning it if that's the case. >> >> The autoscaler will usually only bump pg_num for a pool when the value is >> <half what it thinks it should be. >> >> I suggest setting the ‘bulk’ flag one pool at a time to effectively >> pre-split PGs as if the pool were full of data already. Ignore .mgr. >> Start with the pools with the fewest , let the cluster settle between each >> adjustment. That way the autoscaler will only make changes if cluster >> topology changes. >> >> >> > Thank you both so much! It definitely helped me to understand Ceph >> better. >> > It is kind of a steep curve :). >> >> We’re a community! That curve is way gentler than it used to be. > > > > -- > Bruno Gomes Pessanha > -- Bruno Gomes Pessanha _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |