The controller might do several different things, you should first test how it behaves Does it flush data from cache to disk when instructed to? Most controllers ignore flushes in writeback mode (that is why they are so fast), which is both bad and good - bad when the controller dies but good for performance. And when the controller ignores flushes then disabling barriers won't help you much (if at all). Does it does it really work when you turn the power off suddenly? (turn off the power off during some heavy writes and remove one of the drives in an array before turning it back on to see if it's buggy or sane). Battery relearning/testing should be left enabled. Or does it drain the battery completely? I don't think it does, that would be bad. Another issue is whether disabling barriers keeps the request ordering (in kernel queue and in controller/disk), I believe some assumptions here might be wrong. TL;DR: keep barriers enabled but test both - if there's a large discrepancy then something is fishy.... Jan > On 05 Oct 2015, at 15:32, Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx> wrote: > > Hello, > > We are building a new Ceph cluster and have a few questions regarding the use of write barriers, controler cache, and disk cache (buffer). > > Greg said that barriers should be used (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/002854.html) for data safety which is the default mount option with XFS. > > But what if we're using RAID controlers with battery backed up cache ? > > XFS faq recommends to disable barriers when using battery backuped up RAID controlers AND to make sure to disable disks's cache. Which we are able to do on our PERC controlers (Dell R730xd). (http://xfs.org/index.php/XFS_FAQ#Write_barrier_support). Also we should make sure to disable battery relearning cycles. > > Questions are : > > Is it safe to use the controler cache in front of SAS/SATA data disks ? (Our tests showed 1.95x more read/write IOPS when using cache) > > Is it safe to use the controler cache in front of SSD metadata disks ? (Our tests showed 1.38x more read/write IOPS when using cache). SSD metadata disks are protected from power loss (http://toshiba.semicon-storage.com/us/product/storage-products/enterprise-ssd/px02smb-px02smfxxx.html) > > When using the controler cache (with multiple single drive RAID0 volumes), should we disable disk cache in any scenario ? Should we use barriers or not ? > > It's not clear to me whether the barrier mechanism will apply to the controler cache or through the controler cache up to the physical disks. > > Regards, > > Frederic. > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |