> One server has LSI SAS3008 [0] instead of the Perc H800, > which comes with 512MB RAM + BBU. On most servers latencies are around > 4-12ms (average 6ms), on the system with the LSI controller we see > 20-60ms (average 30ms) latency. Are these reads, writes, or a mixed workload? I would expect an improvement in writes, but 512MB of cache isn't likely to help much on reads with such a large data set. Just as a test, you could removing the battery on one of the H800s to disable the write cache -- or else disable write caching with megaraid or equivalent. On Mon, Apr 19, 2021 at 12:21 PM Nico Schottelius <nico.schottelius@xxxxxxxxxxx> wrote: > > > Good evening, > > I've to tackle an old, probably recurring topic: HBAs vs. Raid > controllers. While generally speaking many people in the ceph field > recommend to go with HBAs, it seems in our infrastructure the only > server we phased in with an HBA vs. raid controller is actually doing > worse in terms of latency. > > For the background: we have many Perc H800+MD1200 [1] systems running with > 10TB HDDs (raid0, read ahead, writeback cache). > One server has LSI SAS3008 [0] instead of the Perc H800, > which comes with 512MB RAM + BBU. On most servers latencies are around > 4-12ms (average 6ms), on the system with the LSI controller we see > 20-60ms (average 30ms) latency. > > Now, my question is, are we doing some inherently wrong with the > SAS3008 or does in fact the cache help to possible reduce seek time? > > We were considering to move more towards LSI HBAs to reduce maintenance > effort, however if we have a factor of 5 in latency between the two > different systems, it might be better to stay on the H800 path for > disks. > > Any input/experiences appreciated. > > Best regards, > > Nico > > [0] > 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) > Subsystem: Dell 12Gbps HBA > Kernel driver in use: mpt3sas > Kernel modules: mpt3sas > > [1] > 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) > Subsystem: Dell PERC H800 Adapter > Kernel driver in use: megaraid_sas > Kernel modules: megaraid_sas > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |