brassow
Attachment:
perf_matrix.pl
Description: Binary data
On Mar 9, 2007, at 11:49 AM, Jonathan E Brassow wrote:
Nope, the first version is just slow. Next version should be coming with RHEL5.X (and should be going upstream), which should be faster.I just wrote up a perl script (which I haven't had a chance to really clean-up yet) that will give performance numbers for various request/transfer sizes. I'm including it at the end.You must have lmbench package installed (for 'lmdd'). Then run: # to give you read performance numbers 'perf_matrix.pl if=<block device>' # to give you write performance numbers 'perf_matrix.pl of=<block device>' # to do multiple runs and discard numbers outside the std deviation # The more iterations you do, the more accurate your results 'perf_matrix.pl if=<block device> iter=5 For more information on the options, do 'perf_matrix.pl -h'.Using the above, you can compare the numbers you're getting from the base device, linear target, mirror target, etc over a wide range of transfer/request sizes.Let's take a look at a couple examples. (Request sizes increase to the right by powers of two starting at 1kiB. Transfer sizes increase by rows by powers of two starting at 1MiB. Results are in MiB/sec): prompt> perf_matrix.pl if=/dev/vg/linear iter=5 #linear reads, 5 iterations w/ results averaged 25.24 28.16 28.82 28.93 28.96 29.25 28.72 26.54 27.39 27.84 28.94 0.00 0.00 30.48 31.57 31.66 31.32 31.89 32.19 31.66 32.00 33.98 34.23 31.93 33.30 0.00 34.00 33.46 33.39 33.12 33.50 34.32 33.57 33.78 34.81 35.03 33.68 34.25 34.68 34.82 34.33 34.32 34.20 34.49 34.89 35.20 35.24 35.39 35.33 35.56 34.94 35.18 35.50 35.37 35.53 35.37 35.54 35.53 35.41 35.60 35.38 35.53 35.54 35.45 35.33 35.72 35.76 35.82 35.81 35.81 35.80 35.81 35.82 35.81 35.84 35.66 35.78 35.76 35.96 35.97 35.87 35.91 35.98 35.99 35.97 35.97 35.98 35.99 35.90 35.96 35.95 36.05 36.05 36.05 36.03 36.03 36.03 36.06 36.08 36.06 36.07 36.07 36.06 36.06 36.10 36.08 36.08 36.08 36.08 36.10 36.08 36.09 36.10 36.11 36.09 36.10 36.11 36.11 36.11 36.11 36.11 36.11 36.11 36.11 36.12 36.12 36.12 36.12 36.12 36.12 36.13 36.12 36.12 36.12 36.12 36.12 36.13 36.12 36.12 36.13 36.13 36.13 36.13prompt> perf_matrix.pl of=/dev/vg/linear iter=5 #linear writes, 5 iterations w/ results averaged 11.74 9.00 31.77 31.82 31.78 31.84 31.93 32.03 32.37 32.98 34.52 0.00 0.00 9.14 9.65 33.57 33.65 33.64 33.65 33.70 33.79 33.99 34.33 35.12 33.36 0.00 9.63 9.70 33.03 33.01 34.65 34.65 34.67 33.09 33.16 33.35 33.70 32.88 34.42 9.60 9.66 33.30 32.35 33.47 33.49 33.49 32.73 33.36 33.65 33.84 33.41 33.37 9.68 9.74 33.31 33.36 32.90 32.94 32.94 33.21 33.08 32.99 33.16 33.33 32.59 9.66 9.74 32.88 33.14 33.47 33.38 33.20 33.60 33.18 33.35 33.15 33.10 33.22 9.68 9.73 32.66 32.73 33.30 33.39 33.22 33.18 33.23 32.97 33.01 33.10 33.13 9.69 9.74 33.06 33.28 33.37 33.45 33.32 33.53 33.27 33.34 33.16 33.05 33.08 9.59 9.66 31.88 32.34 32.14 32.41 33.21 32.49 32.41 32.47 32.39 32.69 32.05 9.47 9.58 32.87 32.79 32.80 32.84 33.09 32.96 32.99 32.95 32.65 32.59 32.83 9.45 9.52 33.35 33.10 33.17 33.12 33.05 33.12 33.97 33.14 32.72 33.07 33.24# if I redirect the above output to files, I can then diff them prompt> perf_matrix.pl diff clinear-read.txt clinear-write.txt-53.49% -68.04% 10.24% 9.99% 9.74% 8.85% 11.18% 20.69% 18.18% 18.46% 19.28% -.--% -.--% -70.01% -69.43% 6.03% 7.44% 5.49% 4.54% 6.44% 5.59% 0.03% 0.29% 9.99% 0.18% -.--% -71.68% -71.01% -1.08% -0.33% 3.43% 0.96% 3.28% -2.04% -4.74% -4.80% 0.06% -4.00% -0.75% -72.43% -71.86% -2.97% -5.41% -2.96% -4.01% -4.86% -7.12% -5.74% -4.76% -4.84% -4.38% -5.14% -72.73% -72.46% -6.25% -5.68% -7.43% -7.29% -6.98% -6.71% -6.50% -7.15% -6.70% -5.98% -7.76% -72.96% -72.76% -8.21% -7.46% -6.53% -6.76% -7.29% -6.20% -7.34% -6.95% -7.04% -7.49% -7.10% -73.08% -72.95% -8.95% -8.86% -7.45% -7.22% -7.65% -7.76% -7.64% -8.39% -8.05% -7.95% -7.84% -73.12% -72.98% -8.29% -7.63% -7.38% -7.16% -7.60% -7.07% -7.74% -7.57% -8.07% -8.35% -8.26% -73.43% -73.23% -11.64% -10.37% -10.92% -10.22% -7.95% -9.98% -10.22% -10.08% -10.25% -9.45% -11.24% -73.77% -73.47% -8.97% -9.19% -9.17% -9.06% -8.36% -8.75% -8.67% -8.78% -9.61% -9.77% -9.11% -73.84% -73.64% -7.67% -8.36% -8.17% -8.31% -8.52% -8.31% -5.95% -8.28% -9.44% -8.47% -8.00%I can see that writes for a linear device are much worse when request sizes are small, but get reasonably close when request sizes are >= 4kiB.I haven't had a chance to do this with (cluster) mirrors yet. It would be interesting to see the difference in performance from linear -> mirror and mirror -> cmirror...Once things are truly stable, I will concentrate more on performance. (Also note: While a mirroring is sync'ing itself, performance for nominal operations will be degraded.)brassow <perf_matrix.pl> On Mar 8, 2007, at 12:13 PM, Robert Clark wrote:I've been trying out cmirror for a few months on a RHEL4U4 cluster andit's now working very well for me, although I've noticed that it does have a bit of a performance hit. My set-up has a 32G GFS filesystem on a mirrored LV shared via AoE (with jumbo frame support). Just using dd with a 4k blocksize to writefiles on the same LV when it's mirrored and then unmirrored shows a bigdifference in speed: Unmirrored: 12440kB/s Mirrored: 2969kB/swhich I wasn't expecting as my understanding is that the cmirror designintroduces very little overhead.The two legs of the mirror are on separate, identical AoE servers andthe filesystem is mounted on 3 out of 6 nodes in the cluster. This iswith the cmirror-kernel_2_6_9_19 tagged version and I've tried with bothcore and disk logs. I suspect a bad interaction between cmirror and something else, but I'm not sure where to start looking. Any ideas? Thanks, Robert -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
