On Tue, Oct 06, 2009 at 03:50:45PM -0400, Alan D. Brunelle wrote: > This looks pretty bad - could you tell me what distro you are running on > as a base? And did this happen before 2.6.30.5 (the poor performance)? > And could you provide a more complete snippet of blktrace output > (showing a handful of complete ops)? Hi, Alan! The poor performance has been ongoing for some time (these boxes were built circa 2.6.26, and we hit a number of issues along the way -- 2.6.30 was the first stable kernel for serving files via nfsd, even with EXT3). I'm figuring the majority of the performance issue are actually with the AOE driver or the Coraid queue scheduling or RAID implementation, but I figured that unplugging by timer maybe shouldn't happen even regardless of how slow the underlying "device" is, hence my email. It's amd64 Debian lenny (not sure why that matters), 16 GB of RAM, about and a whole bunch of AOE storage, chopped up via DM, mostly using XFS as a file system, served via knfsd. "iostat -x -k 1" shows 100% utilization fairly often for this particular device. It's possible that this is just aggravated by memory allocator latency or something, causing the unplug timer to trigger. I was just wondering if this is normal behaviour. blktrace snippets seem pretty massive: [sroot@nas04:/root/trace]# blktrace -w 10 /dev/etherd/e7.0 Device: /dev/etherd/e7.0 CPU 0: 0 events, 843 KiB data CPU 1: 0 events, 709 KiB data CPU 2: 0 events, 712 KiB data CPU 3: 0 events, 752 KiB data CPU 4: 0 events, 714 KiB data CPU 5: 0 events, 713 KiB data CPU 6: 0 events, 619 KiB data CPU 7: 0 events, 732 KiB data Total: 0 events (dropped 0), 5791 KiB data I've saved that here: http://0x.ca/sim/ref/2.6.30.5-hw-fixedxfs/nas04_blktrace_snippet.tgz ...which can be extracted and read with "blkparse etherd_e7.0". Cheers, Simon- -- To unsubscribe from this list: send the line "unsubscribe linux-btrace" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
