Re: New best practices for osds???

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



in most cases write back cache does help a lot for hdd write latency, either raid-0 or some Areca cards support write back in jbod mode. Our observation they could help by a 3-5x factor in Bluestore, whereas db/wal on flash will be about 2x, it does depend on hardware but in general we see benefit in combining both if hdds are used. With Filestore the effect of journal on flash was better than 2x in terms of latency/iops.

/Maged

On 17/07/2019 15:00, Mark Nelson wrote:
Some of the first performance studies we did back at Inktank were looking at RAID-0 vs JBOD setups! :)  You are absolutely right that the controller cache (especially write-back with a battery or supercap) can help with HDD-only configurations.  Where we typically saw problems was when you load up a chassis with lots of drives and use SAS expanders with a single controller.  In some cases we saw higher tail latency and of course you can hit throughput limitations for large IOs too with enough disks.  This happens much quicker if you've got SSDs in the mix for DB/WAL too. Back then, the question really was whether you were better off investing money in a controller+cache or jbod-only setup with flash journals (now DB/WAL).  I'm guessing it's still worth prioritizing flash over the controller, but if you've already got the controller it may not be a bad idea to use single-disk RAID0 depending on your use case.  JBOD does make system management a bit more friendly imho.


Regarding Disk Cache:  We've diagnosed some very strange disk cache behavior with customers in the past.  Nothing recent though.


Mark


On 7/17/19 7:27 AM, John Petrini wrote:
Dell has a whitepaper that compares Ceph performance using JBOD and RAID-0 per disk that recommends RAID-0 for HDD's: en.community.dell.com/techcenter/cloud/m/dell_cloud_resources/20442913/download <http://en.community.dell.com/techcenter/cloud/m/dell_cloud_resources/20442913/download>

After switching from JBOD to RAID-0 we saw a huge reduction in latency, the difference was much more significant than that whitepaper shows. RAID-0 allows us to leverage the controller cache which has major performance improvements when used with HDD's. We also disable the disk cache on our HDD's and SSD's as we had inconsistent performance with disk cache enabled.

As always I'd suggest testing various configurations with your own hardware but I wouldn't shy away from RAID-0 simply because of "best practice".

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux