On Fri, Jan 25, 2013 at 10:26 AM, Blake Hudson <blake@xxxxxxxx> wrote:
I found the problem as I was attempting to set a policy of write back (wb) and it
said success but status showed write through. Then I saw entries in the logs
predating my use of the omconfig command with writepolicy, and that was the clue.
This sample command is a way to dump the controller's log to disk:
/opt/dell/srvadmin/sbin/omconfig storage controller controller=0 action="">
Then look for "Absolute" (at least true for Perc 5/i):
grep Absolute /var/log/lsi_0125.log
T27: Absolute State of Charge : 33 %
T27: Absolute State of Charge : 29 %
T27: Absolute State of Charge : 33 %
T27: Absolute State of Charge : 29 %
T27: Absolute State of Charge : 33 %
T27: Absolute State of Charge : 29 %
I believe at below 30% it is the threshold where write back is disabled.
I'm glad we caught this as there are a number of Dell 2950 systems
in a similar state or about to be.
francis picabia wrote the following on 1/25/2013 7:55 AM:
The PERC cards will disable write-back caching while the BBU is charging/exercising. However, within a few hours the BBU should return to normal status. In rare instances, people on the Dell mailing list have reported that their caching status never returns to write-back - even after attempting to force write-back caching on the array. Attempts and power cycling or firmware flashing are tried, but seem to be futile in most cases. Often, replacement of the card is necessary. I'm unsure if it's the battery, the card, or some software setting, but I would definitely follow up with Dell.
On Thu, Jan 24, 2013 at 12:22 PM, Blake Hudson <blake@xxxxxxxx> wrote:
There are a couple suggestions I'd like to put forth. First, improper
partition alignment is generally masked by the controller cache. I
strongly encourage you to check that your RAID array is making use of
this cache by enabling the WriteBack caching option on this array,
especially if your PERC card has a BBU (I think this was optional on
perc 5). You can install the MegaCLI tool from LSI to verify this (can
also be checked from OpenManage or reboot into the controller BIOS).
Thanks for this tip. It put me on to what is wrong.
Jan 18 07:25:39 myserv Server Administrator: Storage Service EventID: 2335 Controller event log: BBU disabled; changing WB virtual disks to WT: Controller 0 (PERC 5/i Integrated)
Bingo! We had write back all along, and the performance tanked when it
fell back to write through. I was wondering why my policy change attempts
were flipping back when I tried testing WB this morning!
This explains everything we've been seeing. Wow. Gotta call Dell.
Thanks everyone for the assistance. I didn't think a battery which shows OK
status in omreport could wound us!
I found the problem as I was attempting to set a policy of write back (wb) and it
said success but status showed write through. Then I saw entries in the logs
predating my use of the omconfig command with writepolicy, and that was the clue.
This sample command is a way to dump the controller's log to disk:
/opt/dell/srvadmin/sbin/omconfig storage controller controller=0 action="">
Then look for "Absolute" (at least true for Perc 5/i):
grep Absolute /var/log/lsi_0125.log
T27: Absolute State of Charge : 33 %
T27: Absolute State of Charge : 29 %
T27: Absolute State of Charge : 33 %
T27: Absolute State of Charge : 29 %
T27: Absolute State of Charge : 33 %
T27: Absolute State of Charge : 29 %
I believe at below 30% it is the threshold where write back is disabled.
I'm glad we caught this as there are a number of Dell 2950 systems
in a similar state or about to be.
On the next server (or array) you configure, I would attempt to align your partitions as you've investigated. Sector 2048 seems to be a good starting position for most RAID levels. I have no conclusive evidence that a different file system or alignment improves my performance, because I've never done a fair side by side test with controlled inputs. However, we use ext4 and do align our partitions using RAID10 on 15k SAS drives for all our Cyrus installs. I have found some issues with the newer systems that I attribute to the move from ext3 to ext4 which can result in MySQL replication problems on power loss/freeze, but these issues are vary rare and usually easy to recover from in our environment. I also notice that new systems always perform better than the old systems, even with identical hardware - I've often attributed this to fragmentation.
This exercise has been useful to compile a set of things we want to ensure
on future system setups for best performance:
1. RAID 10
2. ext4 (fsync problem gone)
3. align partitions to MB, not cylinders
4. use write-back writepolicy and automate check of the BBU status
(don't ever want to reach a BBU disabled state)
According to Redhat's feature request bugzilla report, converting to ext4 from ext3
is unsupported mash up. They recommend only new ext4 creation.
Thanks to all for their recommendations.
---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
