Hello!
According to LSI2308 performance I have the following interesting observation:
According to LSI2308 performance I have the following interesting observation:
At the time of the initial tests I've made a number of tests and chose Samsung SM863 drives as a core of the all-flash pools and journals. The tests were done on the workstation with Intel Z97 chipset via internal SATA3 port.
And I've got the following result (here is only part of the test, full results can be found in the comment to Sebastian Han's blog post and full explanation of the situation with SAS controller is here):
$ sudo hdparm -W 0 /dev/sdc
$ sudo fio --filename=/dev/sdc --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test
...
write: io=20243MB, bw=345477KB/s, iops=86369, runt= 60001msec
clat (usec): min=28, max=19841, avg=115.18, stdev=42.61
$ sudo hdparm -W 0 /dev/sdc
$ sudo fio --filename=/dev/sdc --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test
...
write: io=20243MB, bw=345477KB/s, iops=86369, runt= 60001msec
clat (usec): min=28, max=19841, avg=115.18, stdev=42.61
...
But after I've put the same SSD into the server with Intel RMS25LB controller (it's a part of the Intel AH2000JF6GKIT) it shows quarter less IOPS and worse latencies (more predictable though):
But after I've put the same SSD into the server with Intel RMS25LB controller (it's a part of the Intel AH2000JF6GKIT) it shows quarter less IOPS and worse latencies (more predictable though):
$ sudo hdparm -W 0 /dev/sdc # actually does nothing
$ echo "temporary write through" | sudo tee /sys/block/sdc/device/scsi_disk/0\:0\:0\:0/cache_type # does the actual trick with cache on rms26lb controller, without it results much worse
$ sudo fio --filename=/dev/sdc --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test
...
$ echo "temporary write through" | sudo tee /sys/block/sdc/device/scsi_disk/0\:0\:0\:0/cache_type # does the actual trick with cache on rms26lb controller, without it results much worse
$ sudo fio --filename=/dev/sdc --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test
...
write: io=14603MB, bw=249210KB/s, iops=62302, runt= 60002msec
clat (usec): min=66, max=947, avg=158.92, stdev=30.05
...
I don't have another compatible SAS controllers around to do more tests and I'm not sure this isn't connected with mis-configuration, but this is what I see. I've run tests on different servers and with different SSD samples - results are the same.
RMS25LBs have the latest firmware (20.0).
By the way, if somebody will point settings I can use to raise IOPS on rms25lb controller - I'll be happy.
---
Best regards,
Vladimir
---
Best regards,
Vladimir
2017-09-20 18:08 GMT+05:00 Vincent Tondellier <tondellier+ml.ceph-users@xxxxxxxxxxx>:
Marc Roos wrote:
> We use these :
> NVDATA Product ID : SAS9207-8i
> Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308
> PCI-Express Fusion-MPT SAS-2 (rev 05)
>
> Does someone by any chance know how to turn on the drive identification
> lights?
Tested with a MegaRAID SAS 2108 / DELL H700 :
megacli -PDList -a0
get the enclosure and drive number :
Enclosure Device ID: 32
Slot Number: 0
megacli -PdLocate -start -physdrv '[32:0]' -a0
>
> -----Original Message-----
> From: Jake Young [mailto:jak3kaj-Re5JQEeQqe8AvxtiuMwx3w@public. ]gmane.org
> Sent: dinsdag 19 september 2017 18:00
> To: Kees Meijs; ceph-users-Qp0mS5GaXlQ@public.gmane.org
> Subject: Re: What HBA to choose? To expand or not to
> expand?
>
>
> On Tue, Sep 19, 2017 at 9:38 AM Kees Meijs
> ceph-users-> <kees-FaqLbeXgz6Y@public.gmane.org > wrote:
>
>
> Hi Jake,
>
> On 19-09-17 15:14, Jake Young wrote:
> > Ideally you actually want fewer disks per server and more
> servers.
> > This has been covered extensively in this mailing list. Rule of
> thumb
> > is that each server should have 10% or less of the capacity of
> your
> > cluster.
>
> That's very true, but let's focus on the HBA.
>
> > I didn't do extensive research to decide on this HBA, it's simply
> what
> > my server vendor offered. There are probably better, faster,
> cheaper
> > HBAs out there. A lot of people complain about LSI HBAs, but I am
> > comfortable with them.
>
> Given a configuration our vendor offered it's about LSI/Avago
> 9300-8i
> with 8 drives connected individually using SFF8087 on a backplane
> (e.g.
> not an expander). Or, 24 drives using three HBAs (6xSFF8087 in
> total)
> when using a 4HE SuperMicro chassis with 24 drive bays.
>
> But, what are the LSI complaints about? Or, are the complaints
> generic
> to HBAs and/or cryptic CLI tools and not LSI specific?
>
>
> Typically people rant about how much Megaraid/LSI support sucks. I've
> been using LSI or MegaRAID for years and haven't had any big problems.
>
> I had some performance issues with Areca onboard SAS chips (non-Ceph
> setup, 4 disks in a RAID10) and after about 6 months of troubleshooting
> with the server vendor and Areca support they did patch the firmware and
> resolve the issue.
>
>
>
>
> > There is a management tool called storcli that can fully
> configure the
> > HBA in one or two command lines. There's a command that
> configures
> > all attached disks as individual RAID0 disk groups. That command
> gets
> > run by salt when I provision a new osd server.
>
> The thread I read was about Areca in JBOD but still able to utilise
> the
> cache, if I'm not mistaken. I'm not sure anymore if there was
> something
> mentioned about BBU.
>
>
> JBOD with WB cache would be nice so you can get smart data directly from
> the disks instead of having interrogate the HBA for the data. This
> becomes more important once your cluster is stable and in production.
>
> IMHO if there is unwritten data in a RAM chip, like when you enable WB
> cache, you really, really need a BBU. This is another nice thing about
> using SSD journals instead of HBAs in WB mode, the journaled data is
> safe on the SSD before the write is acknowledged.
>
>
>
>
> >
> > What many other people are doing is using the least expensive
> JBOD HBA
> > or the on board SAS controller in JBOD mode and then using SSD
> > journals. Save the money you would have spent on the fancy HBA
> for
> > fast, high endurance SSDs.
>
> Thanks! And obviously I'm very interested in other comments as
> well.
>
> Regards,
> Kees
>
> _______________________________________________
> ceph-users mailing list
idqoXFIVOFJgJs9I8MT0rw@public. gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
