Yes, I understand. Many thank Eugen Vào Th 2, 7 thg 10, 2024 vào lúc 16:29 Eugen Block <eblock@xxxxxx> đã viết: > Well, if you have more PGs, the deep-scrubs per PG finish faster, but > you have more PGs to scrub. But that way you can stretch the > deep-scrub interval. As I said, I can not recommend to increase them > in general. Overloading OSDs with too many PGs can have a negative > effect as well. If you can, I'd probably rather add more (smaller) > OSDs to spread the PGs. > > Zitat von Phong Tran Thanh <tranphong079@xxxxxxxxx>: > > > Hi Eugen > > > > Adding PG to the cluster only helps to reduce PG size and reduce scrub > time > > is right? >150 PG and <200 in osd i think it's good number > > > > My cluster I/O 2-3GB/s real time in 24h IOPS 2K. If choosing to increase > PG > > is it a good choice? > > > > Vào Th 2, 7 thg 10, 2024 vào lúc 15:49 Eugen Block <eblock@xxxxxx> đã > > viết: > > > >> So your PGs for pool 52 have a size of around 320 GB, that is quite a > >> lot and not surprising that deep-scrubs take a long time. At the same > >> time, your PGs per OSD are already > 150. We had a similar situation > >> on a customer cluster this year as well, also with 12 TB drives. We > >> decided to increase the pg_num anyway to reduce the pg sizes. They > >> currently have around 380 PGs per large OSD (they have lots of smaller > >> OSDs as well) which still works fine. But they're using it as an > >> archive, so the IO is not very high. If you would decide to split PGs, > >> keep in mind to increase mon_max_pg_per_osd and > >> osd_max_pg_per_osd_hard_ratio as well. I can't explicitly recommend to > >> double your PGs per OSD as I'm not familiar with your cluster, the > >> load etc. It's just something to think about. > >> Doubling the PG count would reduce the PG size to around 160 GB, which > >> is still a lot, but I probably wouldn't go further than that. > >> The OSD utilization is only around 40%, in this case a cluster with > >> more (smaller) OSDs would probably have made more sense. > >> > >> Zitat von Phong Tran Thanh <tranphong079@xxxxxxxxx>: > >> > >> > Hi Eugen > >> > > >> > Can you see and give me some advice, number of PG and PG size.. > >> > > >> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META > >> > AVAIL %USE VAR PGS STATUS > >> > 2 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 5.0 MiB > 12 > >> GiB > >> > 6.5 TiB 40.77 1.00 182 up > >> > 17 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.6 TiB 22 MiB > 14 > >> GiB > >> > 6.1 TiB 44.16 1.08 196 up > >> > 32 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 37 MiB > 12 > >> GiB > >> > 6.7 TiB 38.80 0.95 173 up > >> > 47 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 655 KiB > 11 > >> GiB > >> > 6.8 TiB 38.16 0.93 184 up > >> > 60 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 19 MiB > 12 > >> GiB > >> > 6.6 TiB 39.47 0.96 176 up > >> > 74 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 28 MiB > 12 > >> GiB > >> > 6.8 TiB 38.10 0.93 187 up > >> > 83 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.5 TiB 1.9 MiB > 14 > >> GiB > >> > 6.2 TiB 43.47 1.06 180 up > >> > 96 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 38 MiB > 12 > >> GiB > >> > 6.7 TiB 38.80 0.95 181 up > >> > 110 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 4.3 MiB > 13 > >> GiB > >> > 6.5 TiB 40.79 1.00 174 up > >> > 123 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 1.9 MiB > 13 > >> GiB > >> > 6.8 TiB 38.11 0.93 173 up > >> > 136 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 43 MiB > 12 > >> GiB > >> > 6.6 TiB 39.46 0.96 179 up > >> > ..... > >> > > >> > PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES > OMAP_BYTES* > >> > OMAP_KEYS* LOG LOG_DUPS STATE SINCE > >> > 52.0 80121 0 0 0 323105248 0 > >> > 0 1747 3000 active+clean 94m > >> > 52.1 79751 0 0 0 321711556766 > 0 > >> > 0 1727 3000 active+clean 21h > >> > 52.2 80243 0 0 0 323711878626 > 0 > >> > 0 1618 3000 active+clean 30h > >> > 52.3 79892 0 0 0 322166010020 > 0 > >> > 0 1627 3000 active+clean 9h > >> > 52.4 80267 0 0 0 323708219486 > 0 > >> > 0 1658 3000 active+clean 5h > >> > 52.5 79996 0 0 0 322331504454 > 0 > >> > 0 1722 3000 active+clean 18h > >> > 52.6 80190 0 0 0 323460394402 > 0 > >> > 0 1759 3000 active+clean 15h > >> > 52.7 79998 0 0 0 322769143546 > 0 > >> > 0 1720 3000 active+clean 26h > >> > 52.8 80292 0 0 0 323932173152 > 0 > >> > 0 1691 3000 active+clean 21h > >> > 52.9 79808 0 0 0 321910742702 > 0 > >> > 0 1675 3000 active+clean 7h > >> > 52.a 79751 0 0 0 321578061334 > 0 > >> > 0 1822 3000 active+clean 26h > >> > 52.b 80287 0 0 0 323905164642 > 0 > >> > 0 1793 3000 active+clean 6h > >> > .... > >> > Thanks Eugen > >> > > >> > Vào Th 2, 7 thg 10, 2024 vào lúc 14:45 Eugen Block <eblock@xxxxxx> > đã > >> > viết: > >> > > >> >> Hi, > >> >> > >> >> disabling scrubbing in general is bad idea, because you won't notice > >> >> any data corruption except when it might be too late. > >> >> But you can fine tune scrubbing, for example increase the interval to > >> >> allow fewer scrubs to finish in a longer interval. Or if the client > >> >> load is mainly during business hours, adjust osd_scrub_begin_hour and > >> >> osd_scrub_end_hour to your needs. > >> >> And it also depends on the size of your PGs. The larger the PGs are, > >> >> the longer a deep-scrub would take. So splitting PGs can have a quite > >> >> positive effect in general. Inspect 'ceph osd df' output as well as > >> >> 'ceph pg ls' (BYTES column), you can also share it here if you need > >> >> assistance interpreting those values. > >> >> > >> >> Regards, > >> >> Eugen > >> >> > >> >> Zitat von Phong Tran Thanh <tranphong079@xxxxxxxxx>: > >> >> > >> >> > Hi ceph users! > >> >> > > >> >> > How about the disable scrub and deep-scrub, i want to disable it > >> because > >> >> of > >> >> > its effect on many I/O of my cluster. > >> >> > If I disable scrub how will it affect my cluster? > >> >> > If enabled, scrubbing is not complete and takes a long time. > >> >> > > >> >> > > >> >> > Thank > >> >> > Skype: tranphong079 > >> >> > _______________________________________________ > >> >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >> > >> >> > >> >> _______________________________________________ > >> >> ceph-users mailing list -- ceph-users@xxxxxxx > >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> >> > >> > > >> > > >> > -- > >> > Trân trọng, > >> > > >> > ---------------------------------------------------------------------------- > >> > > >> > *Tran Thanh Phong* > >> > > >> > Email: tranphong079@xxxxxxxxx > >> > Skype: tranphong079 > >> > >> > >> > >> > > > > -- > > Trân trọng, > > > ---------------------------------------------------------------------------- > > > > *Tran Thanh Phong* > > > > Email: tranphong079@xxxxxxxxx > > Skype: tranphong079 > > > > -- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@xxxxxxxxx Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |