Re: Hardware cache settings recomendation

Jan Schermer <jan@xxxxxxxxxxx> · Wed, 17 Jun 2015 09:40:59 +0200

Cache on top of the data drives (not journal) will not help in most cases, those writes are already buffered in the OS - so unless your OS is very light on memory and flushing constantly it will have no effect, it just adds overhead in case a flush comes. I haven’t tested this extensively with Ceph, though.

Cache enabled on journal drive _could_ help if your SSD is very slow (or if you don’t have SSD for journal at all), and if it is large enough (more than the active journal size) it could prolong the life of your SSD - depending on how and when the cache starts to flush. I know from experience that write cache on Areca controller didn't flush at all until it hit a watermark (50% capacity default or something) and it will be faster than some SSDon their own. Some SSD have higher IOPS than the cache can achieve, but you likely won’t saturate that with Ceph.

Another thing is write cache on the drives themselves - I’d leave that on disabled (which is probably the default) unless the drive in question has capacitors to flush the cache in case of power failure. Controllers usually have a whitelist of devices that respect flushes on which the write cache is default=enabled, but in case of for example Dell Perc you would need to have Dell original drives or enable it manually.

YMMV - i’ve hit the controller cache IOPS limit in the past with cheap Dell Perc (H310 was it?) that did ~20K IOPS top on one SSD drive, while the drive itself did close to 40K. On my SSDs, disabling write cache helps latency (good for journal) bud could be troubling for the SSD lifetime.

In any case I don’t think you would saturate either with Ceph, so I recommend you just test the latency with write cache enabled/disabled on the controller and pick the one that gives the best numbers
this is basically how: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/

Ceph recommended way is to use everything as passthrough (initiator/target mode) or JBOD (RAID0 with single drives on some controllers), so I’d stick with that.

Jan

On 17 Jun 2015, at 08:01, Mateusz Skała <mateusz.skala@xxxxxxxxxxx> wrote:

Yes, all disk are in single drive raid 0. Now cache is enabled for all drives, should I disable cache for SSD drives?
Regards,
Mateusz

From: Tyler Bishop [mailto:tyler.bishop@xxxxxxxxxxxxxxxxx] 
Sent: Thursday, June 11, 2015 7:30 PM
To: Mateusz Skała
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  Hardware cache settings recomendation

You want write cache to disk, no write cache for SSD.

I assume all of your data disk are single drive raid 0?

  Tyler Bishop
Chief Executive Officer
513-299-7108 x10
Tyler.Bishop@xxxxxxxxxxxxxxxxx
If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

From: "Mateusz Skała" <mateusz.skala@xxxxxxxxxxx>
To: ceph-users@xxxxxxxxxxxxxx
Sent: Saturday, June 6, 2015 4:09:59 AM
Subject:  Hardware cache settings recomendation

Hi,
Please help me with hardware cache settings on controllers for ceph rbd best performance. All Ceph hosts have one SSD drive for journal.

We are using 4 different controllers, all with BBU: 
·         HP Smart Array P400
·         HP Smart Array P410i
·         Dell PERC 6/i
·         Dell  PERC H700

I have to set cache policy, on Dell settings are:
·         Read Policy 
o   Read-Ahead (current)
o   No-Read-Ahead
o   Adaptive Read-Ahead
·         Write Policy 
o   Write-Back (current)
o   Write-Through 
·         Cache Policy
o   Cache I/O
o   Direct I/O (current)
·         Disk Cache Policy
o   Default (current)
o   Enabled
o   Disabled
On HP controllers:
·         Cache Ratio (current: 25% Read / 75% Write)
·         Drive Write Cache
o   Enabled (current)
o   Disabled

And there is one more setting in LogicalDrive option:
·         Caching: 
o   Enabled (current)
o   Disabled

Please verify my settings and give me some recomendations. 
Best regards,
Mateusz

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com