After lot of digging today finaly figured out that it's not really using PERC controller but some Fusion MPT. Then it wasn't clear which tool it supports. Finally I installed lsiutil and was able to change the cache size. [root at dsdb1 ~]# lspci|grep LSI 02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) [root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=40k oflag=direct 1024+0 records in 1024+0 records out 134217728 bytes (134 MB) copied, 0.742517 seconds, 181 MB/s I compared this with SW RAID mdadm that I created yesterday on one of the servers and I get around 300MB/s. I will test out first with what we have before destroying and testing with mdadm. Thanks for your help that led me to this path. Another question I had was when creating mdadm RAID does it make sense to use multipathing? On Thu, Apr 21, 2011 at 10:30 AM, Joe Landman <landman at scalableinformatics.com> wrote: > In the lab working on fixing a unit will be about an hour > Please pardon brevity and spelling errors, sent from my BlackBerry > > -----Original Message----- > From: Mohit Anchlia <mohitanchlia at gmail.com> > Date: Thu, 21 Apr 2011 10:29:52 > To: <landman at scalableinformatics.com>; <gluster-users at gluster.org> > Subject: Re: Performance > > Hi Joe, > > When you get chance can you please at my mail? It will be helpful to > get your advise. > > > On Wed, Apr 20, 2011 at 6:16 PM, Mohit Anchlia <mohitanchlia at gmail.com> wrote: >> Yes indeed. As soon as I sent you last email I realized that and set >> the write back option. Now I get 130MB/s better but still nowhere >> close to 600MB/s as advertised or what others say one should see. >> >> What are your recommendations about HW choice? What is more >> preferrable and better? >> >> Another question do I need to set WCE on all the disks first before >> creating RAID0? Or can I do that after creating RAID0? I tried to set >> WCE on existing RAID0 but it fails "change_mode_page: failed fetching >> page: Caching (SBC)". >> >> ?sdparm --set=WCE --save /dev/sda >> ? ?/dev/sda: FUJITSU ? MBD2300RC ? ? ? ? D809 >> >> dd if=/dev/zero of=/dev/sda bs=128k count=80k oflag=direct >> 81920+0 records in >> 81920+0 records out >> 10737418240 bytes (11 GB) copied, 82.8041 seconds, 130 MB/s >> >> Thanks for your help as always. >> >> On Wed, Apr 20, 2011 at 5:53 PM, Joe Landman >> <landman at scalableinformatics.com> wrote: >>> On 04/20/2011 07:50 PM, Mohit Anchlia wrote: >>>> >>>> I did that but it looks the same. I did get an error even though it >>>> says write-caching is on. >>>> >>>> [root at dslg1 ~]# hdparm -W1 /dev/sda >>>> >>>> /dev/sda: >>>> ?setting drive write-caching to 1 (on) >>>> ?HDIO_DRIVE_CMD(setcache) failed: Invalid argument >>>> [root at dslg1 ~]# hdparm /dev/sda >>> >>> You might need sdparm >>> >>> ? ? ? ?sdparm -a /dev/sda | grep WCE >>> >>> With WCE on I see >>> >>> ? ? ? ?[root at smash ~]# sdparm -a /dev/sda | grep WCE >>> ? ? ? ? ?WCE ? ? ? ? 1 >>> >>> and with it off, I see >>> >>> ? ? ? ?[root at smash ~]# hdparm -W0 /dev/sda >>> >>> ? ? ? ?/dev/sda: >>> ? ? ? ? setting drive write-caching to 0 (off) >>> ? ? ? ? write-caching = ?0 (off) >>> >>> ? ? ? ?[root at smash ~]# sdparm -a /dev/sda | grep WCE >>> ? ? ? ? ?WCE ? ? ? ? 0 >>> >>> You might need to change WCE using >>> >>> ? ? ? ?sdparm --set=WCE -a /dev/sda >>> >>> or similar ... >>> >>>> >>>> /dev/sda: >>>> ?readonly ? ? = ?0 (off) >>>> ?readahead ? ?= 256 (on) >>>> ?geometry ? ? = 36472/255/63, sectors = 585937500, start = 0 >>>> [root at dslg1 ~]# [A >>>> [root at dslg1 ~]# dd if=/dev/zero of=/dev/sda bs=128k count=1k oflag=direct >>>> 1024+0 records in >>>> 1024+0 records out >>>> 134217728 bytes (134 MB) copied, 8.10005 seconds, 16.6 MB/s >>>> >>>> >>>> On Wed, Apr 20, 2011 at 5:45 PM, Joe Landman >>>> <landman at scalableinformatics.com> ?wrote: >>>>> >>>>> On 04/20/2011 07:28 PM, Mohit Anchlia wrote: >>>>>> >>>>>> dd of=/dev/null if=/dev/sda bs=128k count=80k iflag=direct >>>>>> 81920+0 records in >>>>>> 81920+0 records out >>>>>> 10737418240 bytes (11 GB) copied, 83.8293 seconds, 128 MB/s >>>>> >>>>> Ok, this is closer to what I was expecting (really ~150 MB/s would make >>>>> more >>>>> sense to me, but I can live with 128 MB/s). >>>>> >>>>> The write speed is definitely problematic. ?I am wondering if write cache >>>>> is >>>>> off, and other features are turned off in strange ways. >>>>> >>>>> This is a 2 year old SATA disk >>>>> >>>>> [root at smash ~]# dd if=/dev/zero of=/dev/sda2 bs=128k oflag=direct >>>>> dd: writing `/dev/sda2': No space left on device >>>>> 16379+0 records in >>>>> 16378+0 records out >>>>> 2146798080 bytes (2.1 GB) copied, 20.8322 s, 103 MB/s >>>>> >>>>> Write cache is enabled. ?Turning write cache off (might not be so >>>>> relevant >>>>> for a RAID0), >>>>> >>>>> [root at smash ~]# hdparm -W /dev/sda >>>>> >>>>> /dev/sda: >>>>> ?write-caching = ?1 (on) >>>>> [root at smash ~]# hdparm -W0 /dev/sda >>>>> >>>>> /dev/sda: >>>>> ?setting drive write-caching to 0 (off) >>>>> ?write-caching = ?0 (off) >>>>> >>>>> [root at smash ~]# dd if=/dev/zero of=/dev/sda2 bs=128k oflag=direct >>>>> dd: writing `/dev/sda2': No space left on device >>>>> 16379+0 records in >>>>> 16378+0 records out >>>>> 2146798080 bytes (2.1 GB) copied, 155.636 s, 13.8 MB/s >>>>> >>>>> See if you can do an >>>>> >>>>> ? ? ? ?hdparm -W1 /dev/sda >>>>> >>>>> and see if it has any impact on the write speed. ?If you are using a >>>>> RAID0, >>>>> safety isn't so much on your mind anyway, so you can see if you can >>>>> adjust >>>>> your cache settings. ?If this doesn't work, you might need to get to the >>>>> console and tell it to allow caching. >>>>> >>>>> >>>>> -- >>>>> Joseph Landman, Ph.D >>>>> Founder and CEO >>>>> Scalable Informatics, Inc. >>>>> email: landman at scalableinformatics.com >>>>> web ?: http://scalableinformatics.com >>>>> ? ? ? http://scalableinformatics.com/sicluster >>>>> phone: +1 734 786 8423 x121 >>>>> fax ?: +1 866 888 3112 >>>>> cell : +1 734 612 4615 >>>>> >>> >>> >>> -- >>> Joseph Landman, Ph.D >>> Founder and CEO >>> Scalable Informatics, Inc. >>> email: landman at scalableinformatics.com >>> web ?: http://scalableinformatics.com >>> ? ? ? http://scalableinformatics.com/sicluster >>> phone: +1 734 786 8423 x121 >>> fax ?: +1 866 888 3112 >>> cell : +1 734 612 4615 >>> >> >
