Mandi! Frank Schilder In chel di` si favelave... >> iops : min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929 >> iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926 > That looks horrible. Exactly, horrible. The strange thing is that we came from an homegrown Ceph cluster built using old hardware (HP G6 servers) and spare disks, that perform 'better', or at least perform 'more uniformly' that this 'new'. With these embarassing IOPS, sooner or later we reach the point where performances go to the ground, and sometimes it suffices to launch some 'find' on the filesystem involved... > We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s read or write. Originally, I was also a bit afraid that these disks would drag performance down, but they are on par with the NL-SAS drives. > For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 Mini (Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is equivalent to JBOD mode or pass-through. These controllers have no cache options, if your do, disable all of them. Mode should be write-through. > For your disk type I saw "volatile write cache available = yes" on "the internet". This looks a bit odd, but maybe these HDDs do have some volatile cache. Try to disable it with smartctl and do the benchmark again. Sorry, i'm a bit puzzled here. Matthias suggest to enable write cache, you suggest to disble it... or i'm cache-confused?! ;-) My actually controller configuration is: root@pppve1:~# perccli /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.1910.0000.0000 Oct 08, 2021 Operating system = Linux 5.4.203-1-pve Controller = 0 Status = Success Description = None Product Name = PERC H750 Adapter Serial Number = 23L01Y6 SAS Address = 5f4ee0802ba3a400 PCI Address = 00:b3:00:00 System Time = 04/14/2023 18:03:24 Mfg. Date = 03/25/22 Controller Time = 04/14/2023 16:03:22 FW Package Build = 52.16.1-4405 BIOS Version = 7.16.00.0_0x07100501 FW Version = 5.160.02-3552 Driver Name = megaraid_sas Driver Version = 07.713.01.00-rc1 Current Personality = RAID-Mode Vendor Id = 0x1000 Device Id = 0x10E2 SubVendor Id = 0x1028 SubDevice Id = 0x2176 Host Interface = PCI-E Device Interface = SAS-12G Bus Number = 179 Device Number = 0 Function Number = 0 Domain ID = 0 Security Protocol = None JBOD Drives = 6 JBOD LIST : ========= --------------------------------------------------------------------------------- ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port --------------------------------------------------------------------------------- 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 --------------------------------------------------------------------------------- ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down Physical Drives = 6 PD LIST : ======= ---------------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ---------------------------------------------------------------------------------- 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD 64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD 64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD 64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD ---------------------------------------------------------------------------------- EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded UBUnsp=UBad Unsupported|Rbld=Rebuild Enclosures = 1 Enclosure LIST : ============== -------------------------------------------------------------------- EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific -------------------------------------------------------------------- 64 OK 8 6 0 0 0 0 0 - BP14G+ ? -------------------------------------------------------------------- EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product ID BBU_Info : ======== ---------------------------------------------- Model State RetentionTime Temp Mode MfgDate ---------------------------------------------- BBU Optimal 0 hour(s) 48C - 0/00/00 ---------------------------------------------- So BBU seems enabled. -- Latex fetish implies the existence of a MS Word fetish. (@nucleus) _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |