I don’t have the firmware versions handy, but at one point around the 2014-2015 timeframe I found that both LSI’s firmware and storcli claimed that the default setting was DiskDefault, ie. leave whatever the drive has alone. It turned out, though, that for the 9266 and 9271, at least, behind the scenes it was claiming DiskDefault, but was actually turning on the drive’s volatile cache. Which resulted in the power-loss behavior you describe. There were also hardware and firmware issues that resulted in preserved / pinned cache not being properly restored - in one case, if even a drive failed hard, I had to replace the HBA in order to boot. I posted a list of RoC HBA thoughts including these to the list back around late-summer 2017. > > I don't have any performance bits to offer, but I do have one experiential bit to offer. > > My initial ceph deployment was on existing servers, that had LSI raid controllers (3108 specifically). > We created R0 vd's for each disk, and had BBUs so were using write back caching. > The big problem that arose was the pdcache value, which in my case defaults to on. > > We had a lightning strike that took out the datacenter, and we lost 21/24 OSDs. > Granted, this was back in XFS-on-filestore days, but this was a painful lesson learned. > It was narrowed down to the pdcache and not to the raid controller caching functions after carrying out some power-loss scenarios after the incident. > > So, make sure you turn your pdcache off in perccli. > > Reed > >> On Apr 19, 2021, at 1:20 PM, Nico Schottelius <nico.schottelius@xxxxxxxxxxx> wrote: >> >> >> Good evening, >> >> I've to tackle an old, probably recurring topic: HBAs vs. Raid >> controllers. While generally speaking many people in the ceph field >> recommend to go with HBAs, it seems in our infrastructure the only >> server we phased in with an HBA vs. raid controller is actually doing >> worse in terms of latency. >> >> For the background: we have many Perc H800+MD1200 [1] systems running with >> 10TB HDDs (raid0, read ahead, writeback cache). >> One server has LSI SAS3008 [0] instead of the Perc H800, >> which comes with 512MB RAM + BBU. On most servers latencies are around >> 4-12ms (average 6ms), on the system with the LSI controller we see >> 20-60ms (average 30ms) latency. >> >> Now, my question is, are we doing some inherently wrong with the >> SAS3008 or does in fact the cache help to possible reduce seek time? >> >> We were considering to move more towards LSI HBAs to reduce maintenance >> effort, however if we have a factor of 5 in latency between the two >> different systems, it might be better to stay on the H800 path for >> disks. >> >> Any input/experiences appreciated. >> >> Best regards, >> >> Nico >> >> [0] >> 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) >> Subsystem: Dell 12Gbps HBA >> Kernel driver in use: mpt3sas >> Kernel modules: mpt3sas >> >> [1] >> 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) >> Subsystem: Dell PERC H800 Adapter >> Kernel driver in use: megaraid_sas >> Kernel modules: megaraid_sas >> >> -- >> Sustainable and modern Infrastructures by ungleich.ch >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |