William L. Maltby wrote:
On Thu, 2006-06-08 at 11:33 -0400, Sam Drinkard wrote:
Jim Perrin wrote:
<snip>
Knowledge = power, so pursue that. In the meantime, some things that
*used* to be good "Rules of Thumb" (if not sitting on it) that you might
be alert for as you investigate. Unfortunately, some would demand that
you have a test bed to be sure you can 1) recover if necessary and 2)
see if it really works without killing the end-user attitude and
(potentially) your future (although anyone from the 8088 days... ;-
Definately..
1) Many DBMS claim (and rightfully so) big performance gains if they are
on a "raw partition" rather than residing in a file system. If it's a
whole disk, you won't even need to partition it, Linux and at least one
other *IX support operations without such bothersome things.
2) If data is read predominately sequentially and you are in a FS, make
a huge block size when you make the file system. This has more effect if
the number crunching is input-data-intensive. Concomitantly, HDs with
large cache will contribute substantially to reduced wait time. As you
might surmise, all forms of cache are a total waste if the data (whether
key or base) is totally wrongly sequenced.
Block size might be an option for sure. The machine is defaulted right
now, but it would not be difficult to back stuff, let the disk get
scribbled to and increase block size, then restore. Unfortunately, this
is nowhere like a database, at least from what I know about how it
works, so doubt if DBMS type stuff would apply.
3) Ditto if random reads, but tend to be heavily grouped in consecutive
key ranges. In order for this to be effective, data should occur in most
frequently accessed order and, optimally, the index for that sequence
should be smack in the middle of the data blocks (i.e. first on disk is
appx 50% of the data, then the index with some slack for growth and then
the rest of the data). Better is index on another disk on another IDE
(or whatever) channel). Can everybody say "Bus Mastering"? It's hard to
keep things organized this way on a single disk, but since you're doing
a batch operation, maybe you can make a copy as needed (or there is an
HD backup frequently made?) and operate on that.
There again, it would take more knowledge about the model than I have
available to know what goes on.
Anecdotally demonstrating the effectiveness of the statements about
ordering matching application, in 1988(?) a n00b admin on Oracle
couldn't understand why my app took 1:15 minutes to gen a screen. I
*knew* it wasn't my program, I'm a performance enthusiast. We talked a
bit, I told him to reload his DB in a certain sequence.
Result full screen in about 7 seconds.
To maintain that performance does require periodic reload/reorg.
4) Defrag the appropriate parts occasionally. Whether a file system or
raw partition, stuff must occasionally go into "overflow areas". Unless
your DBMS reorgs/defrags on-the-fly (bad for user response complaints),
the easiest/fastest/cheapest way is cross-HD copies/reloads. After each
cross-HD, scripts direct the apps to the right disk (on start up check a
timestamp and file/part name gened by the reorg process). This only
works if you have a quiescent time.
Not a lot of free time on here. As said before, about 4 hours of idle
time per 24 hours.
5) As Jim (IIRC) mentioned, avoiding access time updates can provide
gains. If you are fortunate enough that no writes need to be done to the
whole partion when you a crunching the chunks (if partition has only the
DB, if not, consider making it so Mr. Spock), remount the partion read
only (mount -o remount,ro <partition-or-label-or-even-mount-point>) for
the duration of the run. This benefits in some VM/kernel internal ways,
to a very small degree too.
Have implemented the noatime, and should know something more about
whether it helps in about 2 hours when this run completes.
There's more, but some may apply to only specific scenarios.
Yep.. maybe some of the folks at the WRF will respond to the tuning
stuff too, or give me some more in depth info about what takes place.
Sam
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
