Scott,
Long story short, yes I agree, the raids are all kinds of wrong and if I had been involved in the build processes they would look very different right now.
I have been playing around with different bs= and count= settings for some simple dd tests tonight and found some striking differences that finally show the new hardware under performing (significantly!) when compared to the old hardware.
That said, we are still struggling to find a postgres-specific test that yields sufficiently different results on old and new hardware that it could serve as an indicator that we had solved the problem prior to shoving this box back into production service and crossing our fingers. The moment we fix the raids, we give away a prime testing scenario. Tomorrow I plan to split the difference and fix the raids on one of the new boxes and not the other.
We are also working on capturing prod logs for playback but that is proving non-trivial due to our existing performance bottleneck and some eccentricities associated with our application.
more to come on this hopefully
many thanks for all of the insights thus far.
On Mon, Mar 4, 2013 at 6:48 PM, Scott Marlowe <scott.marlowe@xxxxxxxxx> wrote:
I'd be more interested in the random results from bonnie++ but my real
world experience tells me that for heavily parallel writes etc a
RAID-10 will stomp a RAID-6 or RAID-60 on the same number of drives.
--
On Mon, Mar 4, 2013 at 4:47 PM, Steven Crandell
<steven.crandell@xxxxxxxxx> wrote:
> Mark,
> I ran pg_fsync_test on log and data LV's on both old and new hardware.
>
> New hardware out performed old on every measurable on the log LV
>
> Same for the data LV's except for the 16kB open_sync write where the old
> hardware edged out the new by a hair (18649 vs 17999 ops/sec)
> and write, fsync, close where they were effectively tied.
>
>
>
>
> On Mon, Mar 4, 2013 at 4:17 PM, John Rouillard <rouilj@xxxxxxxxxxx> wrote:
>>
>> On Mon, Mar 04, 2013 at 03:54:40PM -0700, Steven Crandell wrote:
>> > Here's our hardware break down.
>> >
>> > The logvg on the new hardware is 30MB/s slower (170 MB/s vs 200 MB/s )
>> > than the logvg on the older hardware which was an immediately
>> > interesting
>> > difference but we have yet to be able to create a test scenario that
>> > successfully implicates this slower log speed in our problems. That is
>> > something we are actively working on.
>> >
>> >
>> > Old server hardware:
>> > Manufacturer: Dell Inc.
>> > Product Name: PowerEdge R810
>> > 4x Intel(R) Xeon(R) CPU E7540 @ 2.00GHz
>> > 32x16384 MB 1066 MHz DDR3
>> > Controller 0: PERC H700 - 2 disk RAID-1 278.88 GB rootvg
>> > Controller 1: PERC H800 - 18 disk RAID-6 2,178.00 GB datavg, 4
>> > drive RAID-10 272.25 GB logvg, 2 hot spare
>> > 2x 278.88 GB 15K SAS on controller 0
>> > 24x 136.13 GB 15K SAS on controller 1
>> >
>> > New server hardware:
>> > Manufacturer: Dell Inc.
>> > Product Name: PowerEdge R820
>> > 4x Intel(R) Xeon(R) CPU E5-4620 0 @ 2.20GHz
>> > 32x32 GB 1333 MHz DDR3
>> > Controller 0: PERC H710P - 4 disk RAID-6 557.75 GB rootvg
>> > Controller 1: PERC H810 - 20 disk RAID-60 4,462.00 GB datavg,
>> > 2
>> > disk RAID-1 278.88 GB logvg, 2 hot spare
>> > 28x278.88 GB 15K SAS drives total.
>>
>> Hmm, you went from a striped (raid 1/0) log volume on the old hardware
>> to a non-striped (raid 1) volume on the new hardware. That could
>> explain the speed drop. Are the disks the same speed for the two
>> systems?
>>
>> --
>> -- rouilj
>>
>> John Rouillard System Administrator
>> Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111
>>
>>
>> --
>> Sent via pgsql-performance mailing list (pgsql-performance@xxxxxxxxxxxxxx)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance
>
>
To understand recursion, one must first understand recursion.
