Thanks to Anthony D'Atri, Joshua Beaman and Nino Kotur.
TL;DR- I need to ditch the spinning rust.
As long as all my pools are using all the OSDs (currently necessary)
this is not really a tuning problem- just a consequence of adding awful
old recycled disks to my shiny NVME.
To answer a few questions:
* I've tried KRBD and librbd, also iSCSI, NFS, CephFS on multiple
different physical and virtual OSes, both *nix and Windows.
* I've benchtested with fio and rbd bench.
* Yes I'm using the default replicas and min_size.
* Yes, I already set primary_affinity to zero for the spinning disks.
* No I can't move disks around or add flash disks to the older
machines with the spinning storage in them. Our hardware vendors are
kinda butts.
I have gone back to my hardware vendor to see if they can do a much
better price on more NVME 12 months later. Fingers crossed.
Thanks again for everyone's quick responses!
On 31/05/2023 12:51 am, Thorne Lawler wrote:
Hi folks!
I have a Ceph production 17.2.6 cluster with 6 machines in it - four
newer, faster machines with 4x3.84TB NVME drives each, and two with
24x1.68TB SAS disks each.
I know I should have done something smart with the CRUSH maps for this
up front, but until now I have shied away from CRUSH maps as they
sound really complex.
Right now my cluster's performance, especially write performance, is
not what it needs to be, and I am looking for advice:
1. How should I be structuring my crush map, and why?
2. How does one actually edit and manage a CRUSH map? What /commands/
does one use? This isn't clear at all in the documentation. Are there
any GUI tools out there for managing CRUSH?
3. Is this going to impact production performance or availability
while I'm configuring it? I have tens of thousands of users relying on
this thing, so I can't take any risks.
Thanks in advance!
--
Regards,
Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170
_DDNS
/_*Please note:* The information contained in this email message and any
attached files may be confidential information, and may also be the
subject of legal professional privilege. _If you are not the intended
recipient any use, disclosure or copying of this email is unauthorised.
_If you received this email in error, please notify Discount Domain Name
Services Pty Ltd on 03 9815 6868 to report this matter and delete all
copies of this transmission together with any attachments. /
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
