On Wed, Dec 12, 2012 at 12:29 AM, Gordon Messmer <yinyang@xxxxxxxxx> wrote:
> That may be difficult at this point, because you really want to start by
> measuring the number of IOPS. That's difficult to do if your
> applications demand more than your hardware currently provices.
Since my original posting, we temporarily moved the data from the
centos 5 server to the centos 6 server. We rebuilt the original
(slow) server with centos 6, then migrated the data back. So far
(fingers crossed) so good. I'm running a constant "iostat -kx 30",
and logging it to a file. Disk utilization is virtually always under
50%. Random spikes in the 90% range, but they are few and far
between.
Now that it appears the hardware + software configuration can handle
the load. So I still have the same question: how I can accurately
*quantify* the kind of IO load these servers have? I.e., how to
measure IOPS?
> This might not be the result of your NFS server performance. You might
> actually be seeing bad performance in your directory service. What are
> you using for that service? LDAP? NIS? Are you running nscd or sssd
> on the clients?
Not using a directory service (manually sync'ed passwd files, and
kerberos for authentication). Not running nscd or sssd.
> RAID 6 is good for $/GB, but bad for performance. If you find that your
> performance is bad, RAID10 will offer you a lot more IOPS.
>
> Mixing 15k drives with RAID-6 is probably unusual. Typically 15k drives
> are used when the system needs maximum IOPS, and RAID-6 is used when
> storage capacity is more important than performance.
>
> It's also unusual to see a RAID-6 array with a hot spare. You already
> have two disks of parity. At this point, your available storage
> capacity is only 600GB greater than a RAID-10 configuration, but your
> performance is MUCH worse.
I agree with all that. Problem is, there is a higher risk of storage
failure with RAID-10 compared to RAID-6. We do have good, reliable
*data* backups, but no real hardware backup. Our current service
contract on the hardware is next business day. That's too much down
time to tolerate with this particular system.
As I typed that, I realized we technically do have a hardware
backup---the other server I mentioned. But even the time to restore
from backup would make a lot of people extremely unhappy.
How do most people handle this kind of scenario, i.e. can't afford to
have a hardware failure for any significant length of time? Have a
whole redundant system in place? I would have to "sell" the idea to
management, and for that, I'd need to precisely quantify our situation
(i.e. my initial question).
>> OS is CentOS 5.6, home
>> directory partition is ext3, with options “rw,data=journal,usrquota”.
>
> data=journal actually offers better performance than the default in some
> workloads, but not all. You should try the default and see which is
> better. With a hardware RAID controller that has battery backed write
> cache, data=journal should not perform any better than the default, but
> probably not any worse.
Right, that was mentioned in another response. Unfortunately, I don't
have the ability to test this. My only system is the real production
system. I can't afford the interruption to the users while I fully
unmount and mount the partition (can't change data= type with
remount).
In general, it seems like a lot of IO tuning is "change parameter,
then test". But (1) what test? It's hard to simulate a very
random/unpredictable workload like user home directories, and (2) what
to test on when one only has the single production system? I wish
there were more "analytic" tools where you could simply measure a
number of attributes, and from there, derive the ideal settings and
configuration parameters.
> If your drives are really 4k sectors, rather than the reported 512B,
> then they're not optimal and writes will suffer. The best policy is to
> start your first partition at 1M offset. parted should be aligning
> things well if it's updated, but if your partition sizes (in sectors)
> are divisible by 8, you should be in good shape.
It appears that centos 6 does the 1M offset by default. Centos 5
definitely doesn't do that.
Anyway... as I suggested above, the problem appears to be resolved...
But the "fix" was kind of a shotgun approach, i.e. I changed too many
things at once to know exactly what specific item fixed the problem.
I'm sure this will inevitably come up again at some point, so I'd
still like to learn/understand more to better handle the situation
next time.
Thanks!
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
