screenshot attached showing the IOPs and LAtency from iostat -xtc 2
On 8/14/2020 9:09 AM, Ed Kalk wrote:
ubuntu@ubuntu:/mnt$ sudo fio
--filename=/mnt/sda1/file1.fio:/mnt/sdb1/file2.fio:/mnt/sdc1/file3.fio:/mnt/sdd1/file4.fio:/mnt/sde1/file5.fio:/mnt/sdf1/file6.fio:/mnt/sdg1/file7.fio:/mnt/sdh1/file8.fio:/mnt/sdi1/file9.fio:/mnt/sdj1/file10.fio:/mnt/sdk1/file11.fio:/mnt/sdl1/file12.fio
--size=2GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio
--iodepth=256 --runtime=500 --numjobs=12 --time_based
--group_reporting --name=iops-test-job
^generate fio to all disks
-----
# iostat -xtc 2
^monitor IOPs to all disks
-----
^how to test “fio” to multiple disks and monitor via “iostat -xtc 2“.
I booted via live CD, made test partitions, file systems and mounted.
disks took a while to initialize and settle in after filesystem
create. monitored via iostat till all disks were quiet. then ran fio
to test.
we observed the fist 6 disks (seagate constellation v1 SED) have a
problem, but next 6 don’t. (seagate constellation v2 non-SED)
The “w_await” shows latency is 100x on v1’s, this was a reproduction
of how they performed in ceph.
***suspecting disk firmware, will try updating them.
-Edward Kalk
On Aug 9, 2020, at 10:44 AM, Edward kalk <ekalk@xxxxxxxxxx> wrote:
Thank you for the reply Anthony.
Yes the R510 is of a previous gen. The H700 isn’t my top suspect
though. It’s been clocked to pull 15,000+ iops (with SSD). I’m
familiar with bottlenecking an hba with 24 disk chassis. I only have 6
ceph OSDs on the H700. (this system should be overkill for that.
(ejected all other disks, only 8 inserted)
I’ll try investigating for FW updates on disks and hba.
will also take another look at disk health via smart. they are
repurposed disks and have seen action to say the least. thanks.
-Ed
>>>>>
>>>>>> On Aug 8, 2020, at 6:56 PM, Anthony D'Atri
<anthony.datri@xxxxxxxxx> wrote:
>>>>>>
>>>>>> Ugh, that looks like a pretty old HBA (and system).
>>>>>>
>>>>>> On an R730xd I found an H330 mini to be something of a
bottleneck with 24x SATA SSD.
>>>>>>
>>>>>> o Check your drives for available firmware updates
>>>>>>
>>>>>> o Check your HBA for a firmware update
>>>>>>
>>>>>> o Check that your HBA’s firmware is supported with your
kernel’s driver version — there should be a matrix at Dell somewhere
>>>>>>
>>>>>> o If you have a BBU/supercap, check its status (perccli /c0 … )
>>>>>>
>>>>>> o Check dmesg/logs for HBA errors
>>>>>>
>>>>>> o Check smartctl -a output on the drives, look for wear,
reallocated sectors, errors
>>>>>>
>>>>>>>> On Aug 8, 2020, at 2:55 PM, Edward kalk <ekalk@xxxxxxxxxx> wrote:
>>>>>>> R0, the controller config. (faux-JBOD)
>>>>>>> Single disk group. NoN-Raid.
>>>>>>>
>>>>>>> -Ed
>>>>>>>
>>>>>>>>> On Aug 8, 2020, at 9:20 AM, Anthony D'Atri
<anthony.datri@xxxxxxxxx> wrote:
>>>>>>>> By R0 you mean hdparm ?
>>>>>>>>
>>>>>>>>> On Aug 8, 2020, at 3:49 AM, Edward kalk <ekalk@xxxxxxxxxx>
wrote:
>>>>>>>>>
>>>>>>>>> Im getting poor performance with 5 of my OSDs, Seagate
Constellation ES SED (1) 10k SAS 2TB 3.5 drives.
>>>>>>>>> disk write latency keeps drifting high ,
>>>>>>>>> 100ms-230ms on writes. the other 30 OSDs are performing
well. avg latency 10-20ms
>>>>>>>>>
>>>>>>>>> We observe stats via “iostat -xtc 2” on CEPH server. w_await
showing writes to disk latency.
>>>>>>>>>
>>>>>>>>> **anyone else have poor latency on SEDs?
>>>>>>>>>
>>>>>>>>> The disks were included on a server we acquired. Didn’t know
they would be SEDs.
>>>>>>>>> I’ve read that seagate SED vs. non SED shouldn’t perform
differently, but looks like they are awful. All the SEDs are in the
newly added CEPH node. A Dell R510 with H700 raid card. all disks in
entire cluster are R0, direct write to disk, no cache.
>>>>>>>>> The new server and disk control spec out at about 10x the
ability and IOPs of other servers, so I suspect the SEDs.
>>>>>>>>>
>>>>>>>>> -Ed
>>>>>>>>> _______________________________________________
>>>>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>> --
>>> Thank you for your time,
>>>
>>>
>>>
>>> Edward H. Kalk IV
>>> Information Technology Dept.
>>> Server Specialist
>>> Datacenter Virtualization and Storage Systems
>>> Socket Telecom, LLC.
>>> 2703 Clark Lane
>>> Columbia, MO 65202
>>> 573-817-0000 or 800-socket3 X218
>> --
--
Thank you for your time,
Edward H. Kalk IV
Information Technology Dept.
Server Specialist
Datacenter Virtualization and Storage Systems
Socket Telecom, LLC.
2703 Clark Lane
Columbia, MO 65202
573-817-0000 or 800-socket3 X218
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
