Mandi! Anthony D'Atri In chel di` si favelave... > Dell???s CLI guide describes setting individual drives in Non-RAID, which *smells* like passthrough, not the more-complex RAID0 workaround we had to do before passthrough. > https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-drive-state-commands?guid=guid-d4750845-1f57-434c-b4a9-935876ee1a8e&lang=en-us??? Exactly. NonRAID 'smells' also to us more like passthrough. The firs strange things came from the fact that SATA/SSD disk are passed in a way that seems 'fully transparent' (eg, linux see even the disk S/N), while SAS disk are passed more like an RAID0 disk, with a different S/N. > Spinners are slow, this is news? ;-) > That said, how slow is slow? Testing commands and results or it didn???t happen. A test done some month ago: root@pppve1:~# fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=128k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=hdd-rw-128 hdd-rw-128: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256 ... fio-3.12 Starting 4 processes Jobs: 4 (f=4): [m(4)][0.0%][eta 08d:17h:11m:29s] hdd-rw-128: (groupid=0, jobs=4): err= 0: pid=26198: Wed May 18 19:11:04 2022 read: IOPS=84, BW=10.5MiB/s (11.0MB/s)(1279MiB/121557msec) slat (usec): min=4, max=303887, avg=23029.19, stdev=61832.29 clat (msec): min=1329, max=6673, avg=4737.71, stdev=415.84 lat (msec): min=1543, max=6673, avg=4760.74, stdev=420.10 clat percentiles (msec): | 1.00th=[ 2802], 5.00th=[ 4329], 10.00th=[ 4463], 20.00th=[ 4530], | 30.00th=[ 4597], 40.00th=[ 4665], 50.00th=[ 4732], 60.00th=[ 4799], | 70.00th=[ 4866], 80.00th=[ 4933], 90.00th=[ 5134], 95.00th=[ 5336], | 99.00th=[ 5805], 99.50th=[ 6007], 99.90th=[ 6342], 99.95th=[ 6409], | 99.99th=[ 6611] bw ( KiB/s): min= 256, max= 5120, per=25.18%, avg=2713.08, stdev=780.45, samples=929 iops : min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929 write: IOPS=87, BW=10.9MiB/s (11.5MB/s)(1328MiB/121557msec); 0 zone resets slat (usec): min=9, max=309914, avg=23025.13, stdev=61676.77 clat (msec): min=1444, max=13086, avg=6943.12, stdev=2068.26 lat (msec): min=1543, max=13086, avg=6966.15, stdev=2069.28 clat percentiles (msec): | 1.00th=[ 2769], 5.00th=[ 4597], 10.00th=[ 4799], 20.00th=[ 5067], | 30.00th=[ 5403], 40.00th=[ 5873], 50.00th=[ 6409], 60.00th=[ 7148], | 70.00th=[ 8020], 80.00th=[ 9060], 90.00th=[10134], 95.00th=[10671], | 99.00th=[11610], 99.50th=[11879], 99.90th=[12550], 99.95th=[12550], | 99.99th=[12684] bw ( KiB/s): min= 256, max= 5376, per=24.68%, avg=2762.20, stdev=841.30, samples=926 iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926 cpu : usr=0.05%, sys=0.09%, ctx=2847, majf=0, minf=49 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=10233,10627,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s), io=1279MiB (1341MB), run=121557-121557msec WRITE: bw=10.9MiB/s (11.5MB/s), 10.9MiB/s-10.9MiB/s (11.5MB/s-11.5MB/s), io=1328MiB (1393MB), run=121557-121557msec Disk stats (read/write): sdc: ios=10282/10601, merge=0/0, ticks=3041312/27373721, in_queue=30373472, util=99.99% > Also, firmware matters. Run Dell???s DSU. Controller does not have the latest-latest firmware, but a decent new one; i've looked at chaneglogs and found nothing that seems relevant to performance trouble. Indeed, i'll do an upgrade ASAP. > Give us details, perccli /c0 show, test results etc. root@pppve1:~# perccli /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.1910.0000.0000 Oct 08, 2021 Operating system = Linux 5.4.203-1-pve Controller = 0 Status = Success Description = None Product Name = PERC H750 Adapter Serial Number = 23L01Y6 SAS Address = 5f4ee0802ba3a400 PCI Address = 00:b3:00:00 System Time = 04/11/2023 12:11:50 Mfg. Date = 03/25/22 Controller Time = 04/11/2023 10:11:47 FW Package Build = 52.16.1-4405 BIOS Version = 7.16.00.0_0x07100501 FW Version = 5.160.02-3552 Driver Name = megaraid_sas Driver Version = 07.713.01.00-rc1 Current Personality = RAID-Mode Vendor Id = 0x1000 Device Id = 0x10E2 SubVendor Id = 0x1028 SubDevice Id = 0x2176 Host Interface = PCI-E Device Interface = SAS-12G Bus Number = 179 Device Number = 0 Function Number = 0 Domain ID = 0 Security Protocol = None JBOD Drives = 6 JBOD LIST : ========= --------------------------------------------------------------------------------- ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port --------------------------------------------------------------------------------- 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 --------------------------------------------------------------------------------- ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down Physical Drives = 6 PD LIST : ======= ---------------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ---------------------------------------------------------------------------------- 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD 64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD 64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD 64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD ---------------------------------------------------------------------------------- EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded UBUnsp=UBad Unsupported|Rbld=Rebuild Enclosures = 1 Enclosure LIST : ============== -------------------------------------------------------------------- EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific -------------------------------------------------------------------- 64 OK 8 6 0 0 0 0 0 - BP14G+ ? -------------------------------------------------------------------- EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product ID BBU_Info : ======== ---------------------------------------------- Model State RetentionTime Temp Mode MfgDate ---------------------------------------------- BBU Optimal 0 hour(s) 47C - 0/00/00 ---------------------------------------------- -- La tua tana sta in collina, Dr. Dobermann, seimilioni a metro quadro e tua moglie sembra proprio una regina, pero` e` la moglie di un ladro. (F. De Gregori) _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |