Well, if you have more PGs, the deep-scrubs per PG finish faster, but
you have more PGs to scrub. But that way you can stretch the
deep-scrub interval. As I said, I can not recommend to increase them
in general. Overloading OSDs with too many PGs can have a negative
effect as well. If you can, I'd probably rather add more (smaller)
OSDs to spread the PGs.
Zitat von Phong Tran Thanh <tranphong079@xxxxxxxxx>:
Hi Eugen
Adding PG to the cluster only helps to reduce PG size and reduce scrub time
is right? >150 PG and <200 in osd i think it's good number
My cluster I/O 2-3GB/s real time in 24h IOPS 2K. If choosing to increase PG
is it a good choice?
Vào Th 2, 7 thg 10, 2024 vào lúc 15:49 Eugen Block <eblock@xxxxxx> đã
viết:
So your PGs for pool 52 have a size of around 320 GB, that is quite a
lot and not surprising that deep-scrubs take a long time. At the same
time, your PGs per OSD are already > 150. We had a similar situation
on a customer cluster this year as well, also with 12 TB drives. We
decided to increase the pg_num anyway to reduce the pg sizes. They
currently have around 380 PGs per large OSD (they have lots of smaller
OSDs as well) which still works fine. But they're using it as an
archive, so the IO is not very high. If you would decide to split PGs,
keep in mind to increase mon_max_pg_per_osd and
osd_max_pg_per_osd_hard_ratio as well. I can't explicitly recommend to
double your PGs per OSD as I'm not familiar with your cluster, the
load etc. It's just something to think about.
Doubling the PG count would reduce the PG size to around 160 GB, which
is still a lot, but I probably wouldn't go further than that.
The OSD utilization is only around 40%, in this case a cluster with
more (smaller) OSDs would probably have made more sense.
Zitat von Phong Tran Thanh <tranphong079@xxxxxxxxx>:
> Hi Eugen
>
> Can you see and give me some advice, number of PG and PG size..
>
> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> AVAIL %USE VAR PGS STATUS
> 2 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 5.0 MiB 12
GiB
> 6.5 TiB 40.77 1.00 182 up
> 17 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.6 TiB 22 MiB 14
GiB
> 6.1 TiB 44.16 1.08 196 up
> 32 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 37 MiB 12
GiB
> 6.7 TiB 38.80 0.95 173 up
> 47 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 655 KiB 11
GiB
> 6.8 TiB 38.16 0.93 184 up
> 60 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 19 MiB 12
GiB
> 6.6 TiB 39.47 0.96 176 up
> 74 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 28 MiB 12
GiB
> 6.8 TiB 38.10 0.93 187 up
> 83 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.5 TiB 1.9 MiB 14
GiB
> 6.2 TiB 43.47 1.06 180 up
> 96 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 38 MiB 12
GiB
> 6.7 TiB 38.80 0.95 181 up
> 110 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 4.3 MiB 13
GiB
> 6.5 TiB 40.79 1.00 174 up
> 123 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 1.9 MiB 13
GiB
> 6.8 TiB 38.11 0.93 173 up
> 136 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 43 MiB 12
GiB
> 6.6 TiB 39.46 0.96 179 up
> .....
>
> PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES*
> OMAP_KEYS* LOG LOG_DUPS STATE SINCE
> 52.0 80121 0 0 0 323105248 0
> 0 1747 3000 active+clean 94m
> 52.1 79751 0 0 0 321711556766 0
> 0 1727 3000 active+clean 21h
> 52.2 80243 0 0 0 323711878626 0
> 0 1618 3000 active+clean 30h
> 52.3 79892 0 0 0 322166010020 0
> 0 1627 3000 active+clean 9h
> 52.4 80267 0 0 0 323708219486 0
> 0 1658 3000 active+clean 5h
> 52.5 79996 0 0 0 322331504454 0
> 0 1722 3000 active+clean 18h
> 52.6 80190 0 0 0 323460394402 0
> 0 1759 3000 active+clean 15h
> 52.7 79998 0 0 0 322769143546 0
> 0 1720 3000 active+clean 26h
> 52.8 80292 0 0 0 323932173152 0
> 0 1691 3000 active+clean 21h
> 52.9 79808 0 0 0 321910742702 0
> 0 1675 3000 active+clean 7h
> 52.a 79751 0 0 0 321578061334 0
> 0 1822 3000 active+clean 26h
> 52.b 80287 0 0 0 323905164642 0
> 0 1793 3000 active+clean 6h
> ....
> Thanks Eugen
>
> Vào Th 2, 7 thg 10, 2024 vào lúc 14:45 Eugen Block <eblock@xxxxxx> đã
> viết:
>
>> Hi,
>>
>> disabling scrubbing in general is bad idea, because you won't notice
>> any data corruption except when it might be too late.
>> But you can fine tune scrubbing, for example increase the interval to
>> allow fewer scrubs to finish in a longer interval. Or if the client
>> load is mainly during business hours, adjust osd_scrub_begin_hour and
>> osd_scrub_end_hour to your needs.
>> And it also depends on the size of your PGs. The larger the PGs are,
>> the longer a deep-scrub would take. So splitting PGs can have a quite
>> positive effect in general. Inspect 'ceph osd df' output as well as
>> 'ceph pg ls' (BYTES column), you can also share it here if you need
>> assistance interpreting those values.
>>
>> Regards,
>> Eugen
>>
>> Zitat von Phong Tran Thanh <tranphong079@xxxxxxxxx>:
>>
>> > Hi ceph users!
>> >
>> > How about the disable scrub and deep-scrub, i want to disable it
because
>> of
>> > its effect on many I/O of my cluster.
>> > If I disable scrub how will it affect my cluster?
>> > If enabled, scrubbing is not complete and takes a long time.
>> >
>> >
>> > Thank
>> > Skype: tranphong079
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
>
> --
> Trân trọng,
>
----------------------------------------------------------------------------
>
> *Tran Thanh Phong*
>
> Email: tranphong079@xxxxxxxxx
> Skype: tranphong079
--
Trân trọng,
----------------------------------------------------------------------------
*Tran Thanh Phong*
Email: tranphong079@xxxxxxxxx
Skype: tranphong079
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
