On 2012-06-14, at 10:20 PM, Carlos Carvalho wrote: > Our server has suddenly become extremely slow in writes. With > > % dd if=/dev/zero of=zero bs=2M count=2000 > > I get only 1.4MB/s in an almost idle machine. It has a raid6 with 6 > disks, running 3.3.7. 3.4.2 doesn't improve matters. > > The important point is that it became so slow yesterday. No hardware > has changed and all disks are fine. Reading from the filesystem is > unaffected, at more than 85MB/s. The problem happens both in the root > and home partitions. > > During the dd the disk utilization measured by sar is small: > > DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util > sdb 93.30 254.40 931.20 12.71 0.57 6.10 4.16 38.77 > sdc 100.50 244.00 970.40 12.08 0.76 7.59 3.80 38.20 > sdd 101.40 261.60 927.20 11.72 0.68 6.68 3.74 37.90 > sde 86.80 300.00 780.80 12.45 0.69 7.93 4.29 37.20 > sdf 82.90 315.20 810.40 13.58 0.55 6.60 4.39 36.37 > sda 96.70 220.00 984.00 12.45 0.47 4.87 3.72 35.94 > > So it seems that it's not a disk problem. Both filesystems reached 90% > usage in space but less than 10% in inodes shortly before the slowness > appeared. Now they're at 57% and 77% but still crawl. Could this be > related? Definitely yes. I was going to ask this question even before I got to the end and saw your comment. If the filesystem gets too full, it can cause new allocations to become fragmented, and then even if you delete other files, the fragmented files end up leaving individual allocated blocks all around your filesystem, making future allocations bad also. You can run "e2freefrag" on the block device to report the current sizes for free space in the filesystem. There is the "e4defrag" tool that can defragment files, but I don't recall whether this is working well or not. Cheers, Andreas -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
