Hi Sven,
thanks for the input.
So I did some testing and "maybe" optimization.
The same disk type in two different hosts (one Ubuntu and one Centos7) have
VERY different iostat %util values:
Ubuntu:
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s
wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm
d_await dareq-sz aqu-sz %util
sdh 45.60 2419.20 20.40 30.91 1.03 53.05 1949.40
12928.80 516.00 20.93 0.10 6.63 0.00 0.00 0.00
0.00 0.00 0.00 0.02 66.00
Centos:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdy 6,40 352,80 30,60 1264,40 1705,60 11061,60 19,72
0,24 0,18 2,39 0,13 0,08 9,82
What I tried on the Ubuntu Host:
- disable write cache (no difference)
- set the scheduler from mq-deadline to none (ubuntu seems to not have
noop, or it might be the same) (no difference)
- set tune-adm to latency=performance (made it slightly better)
It is really strange to see these high util values only on the ubuntu host.
Am Di., 6. Dez. 2022 um 19:22 Uhr schrieb Sven Kieske <S.Kieske@xxxxxxxxxxx
>:
> On Di, 2022-12-06 at 15:38 +0100, Boris Behrens wrote:
> > I've cross checked the other 8TB disks in our cluster, which are around
> > 30-50% with roughly the same IOPs.
> > Maybe I am missing some optimization, that is done on the centos7 nodes,
> > but not on the ubuntu20.04 node. (If you know something from the top of
> > your head, I am happy to hear it).
> > Maybe it is just another measuring on ubuntu.
>
> the very first thing I would check is the drive read/write caches:
>
>
> https://docs.ceph.com/en/quincy/start/hardware-recommendations/#write-caches
>
> (this part of the docs also applies to earlier ceph releases, but wasn't
> available
> in older releases)
>
> I'd recommend installing the udev rule which switches write caches off.
>
> You might want to evaluate first if your drives perform better or worse
> without caches.
>
> IIRC there where some reports on this ML that performance was even worse
> on some drives without the cache for certain workloads.
>
> But I never experienced this myself.
>
> HTH
>
> --
> Mit freundlichen Grüßen / Regards
>
> Sven Kieske
> Systementwickler / systems engineer
>
>
> Mittwald CM Service GmbH & Co. KG
> Königsberger Straße 4-6
> 32339 Espelkamp
>
> Tel.: 05772 / 293-900
> Fax: 05772 / 293-333
>
> https://www.mittwald.de
>
> Geschäftsführer: Robert Meyer, Florian Jürgens
>
> St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
> Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
>
> Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit
> gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
