On Mon, May 07, 2012 at 08:39:13AM +0200, Stefan Priebe - Profihost AG wrote: > Hi, > > after deleting 400GB it was faster. Now there are still 300GB free but > it is slow as hell again ;-( > > Am 07.05.2012 03:34, schrieb Dave Chinner: > > On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote: > >> Hi, > >> > >> since a few days i've experienced a really slow fs on one of our > >> backup systems. > >> > >> I'm not sure whether this is XFS related or related to the > >> Controller / Disks. > >> > >> It is a raid 10 of 20 SATA Disks and i can only write to them with > >> about 700kb/s while doing random i/o. > > > > What sort of random IO? size, read, write, direct or buffered, data > > or metadata, etc? > There are 4 rsync processes running and doing backups of other severs. > > > iostat -x -d -m 5 and vmstat 5 traces would be > > useful to see if it is your array that is slow..... > > ~ # iostat -x -d -m 5 > Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_ > (8 CPU) > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06 > sda 0,00 0,20 0,00 1,20 0,00 0,00 6,50 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50 > sda 0,00 0,00 0,00 0,40 0,00 0,00 4,50 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 11,20 242,40 92,00 1,56 0,89 15,00 4,70 14,06 1,58 52,68 > sda 0,00 0,20 0,00 2,60 0,00 0,02 12,00 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56 > sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02 > sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > # vmstat "vmstat 5", not vmstat 5 times.... :/ > >> I tried vanilla Kernel 3.0.30 > >> and 3.3.4 - no difference. Writing to another partition on another > >> xfs array works fine. > >> > >> Details: > >> #~ df -h > >> /dev/sdb1 4,6T 4,4T 207G 96% /mnt > > > > Your filesystem is near full - the allocation algorithms definitely > > slow down as you approach ENOSPC, and IO efficiency goes to hell > > because of a lack of contiguous free space to allocate from. > I've now 94% used but it is still slow. It seems it was just getting > fast with more than 450GB free space. > > /dev/sdb1 4,6T 4,3T 310G 94% /mnt Well, you've probably badly fragmented the free space you have. what does the 'xfs_db -r -c freesp <dev>' command tell you? > >> #~ df -i > >> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt > > You have 4.6 *billion* inodes in your filesystem? > Yes - it backups around 100 servers with a lot of files. So you have what - lots of symlinks? I mean, 4.6 billion inodes alone requires 1.2TB of space, but if I read the fragmentation you only have 82 million files with data extents. The only thing that would other wise use inodes are directories and symlinks.... Still, I can't see how you'd only have 82 million data inodes and 4.5 billion directory inodes - where are all the inodes being consumed? A massive symlink farm? Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
