2012/2/29 Jacek Luczak <difrost.kernel@xxxxxxxxx>: > 2012/2/29 Jacek Luczak <difrost.kernel@xxxxxxxxx>: >> Hi Chris, >> >> the last one was borked :) Please check this one. >> >> -jacek >> >> 2012/2/29 Jacek Luczak <difrost.kernel@xxxxxxxxx>: >>> Hi All, >>> >>> /*Sorry for sending incomplete email, hit wrong button :) I guess I >>> can't use Gmail */ >>> >>> Long story short: We've found that operations on a directory structure >>> holding many dirs takes ages on ext4. >>> >>> The Question: Why there's that huge difference in ext4 and btrfs? See >>> below test results for real values. >>> >>> Background: I had to backup a Jenkins directory holding workspace for >>> few projects which were co from svn (implies lot of extra .svn dirs). >>> The copy takes lot of time (at least more than I've expected) and >>> process was mostly in D (disk sleep). I've dig more and done some >>> extra test to see if this is not a regression on block/fs site. To >>> isolate the issue I've also performed same tests on btrfs. >>> >>> Test environment configuration: >>> 1) HW: HP ProLiant BL460 G6, 48 GB of memory, 2x 6 core Intel X5670 HT >>> enabled, Smart Array P410i, RAID 1 on top of 2x 10K RPM SAS HDDs. >>> 2) Kernels: All tests were done on following kernels: >>> - 2.6.39.4-3 -- the build ID (3) is used here for internal tacking of >>> config changes mostly. In -3 we've introduced ,,fix readahead pipeline >>> break caused by block plug'' patch. Otherwise it's pure 2.6.39.4. >>> - 3.2.7 -- latest kernel at the time of testing (3.2.8 has been >>> release recently). >>> 3) A subject of tests, directory holding: >>> - 54GB of data (measured on ext4) >>> - 1978149 files >>> - 844008 directories >>> 4) Mount options: >>> - ext4 -- errors=remount-ro,noatime, >>> data=writeback >>> - btrfs -- noatime,nodatacow and for later investigation on >>> copression effect: noatime,nodatacow,compress=lzo >>> >>> In all tests I've been measuring time of execution. Following tests >>> were performed: >>> - find . -type d >>> - find . -type f >>> - cp -a >>> - rm -rf >>> >>> Ext4 results: >>> | Type | 2.6.39.4-3 | 3.2.7 >>> | Dir cnt | 17m 40sec | 11m 20sec >>> | File cnt | 17m 36sec | 11m 22sec >>> | Copy | 1h 28m | 1h 27m >>> | Remove| 3m 43sec | 3m 38sec >>> >>> Btrfs results (without lzo comression): >>> | Type | 2.6.39.4-3 | 3.2.7 >>> | Dir cnt | 2m 22sec | 2m 21sec >>> | File cnt | 2m 26sec | 2m 23sec >>> | Copy | 36m 22sec | 39m 35sec >>> | Remove| 7m 51sec | 10m 43sec >>> >>> From above one can see that copy takes close to 1h less on btrfs. I've >>> done strace counting times of calls, results are as follows (from >>> 3.2.7): >>> 1) Ext4 (only to elements): >>> % time seconds usecs/call calls errors syscall >>> ------ ----------- ----------- --------- --------- ---------------- >>> 57.01 13.257850 1 15082163 read >>> 23.40 5.440353 3 1687702 getdents >>> 6.15 1.430559 0 3672418 lstat >>> 3.80 0.883767 0 13106961 write >>> 2.32 0.539959 0 4794099 open >>> 1.69 0.393589 0 843695 mkdir >>> 1.28 0.296700 0 5637802 setxattr >>> 0.80 0.186539 0 7325195 stat >>> >>> 2) Btrfs: >>> % time seconds usecs/call calls errors syscall >>> ------ ----------- ----------- --------- --------- ---------------- >>> 53.38 9.486210 1 15179751 read >>> 11.38 2.021662 1 1688328 getdents >>> 10.64 1.890234 0 4800317 open >>> 6.83 1.213723 0 13201590 write >>> 4.85 0.862731 0 5644314 setxattr >>> 3.50 0.621194 1 844008 mkdir >>> 2.75 0.489059 0 3675992 1 lstat >>> 1.71 0.303544 0 5644314 llistxattr >>> 1.50 0.265943 0 1978149 utimes >>> 1.02 0.180585 0 5644314 844008 getxattr >>> >>> On btrfs getdents takes much less time which prove the bottleneck in >>> copy time on ext4 is this syscall. In 2.6.39.4 it shows even less time >>> for getdents: >>> % time seconds usecs/call calls errors syscall >>> ------ ----------- ----------- --------- --------- ---------------- >>> 50.77 10.978816 1 15033132 read >>> 14.46 3.125996 1 4733589 open >>> 7.15 1.546311 0 5566988 setxattr >>> 5.89 1.273845 0 3626505 lstat >>> 5.81 1.255858 1 1667050 getdents >>> 5.66 1.224403 0 13083022 write >>> 3.40 0.735114 1 833371 mkdir >>> 1.96 0.424881 0 5566988 llistxattr >>> >>> >>> Why so huge difference in the getdents timings? >>> >>> -Jacek > > I will try to answer the question from the broken email I've sent. > > @Lukas, it was always a fresh FS on top of LVM logical volume. I've > been cleaning cache/remounting to sync all data before (re)doing > tests. > > -Jacek > > BTW: Sorry for the email mixture. I just can't get this gmail thing to > work (why forcing top posting:/). Please use this thread. More from the observations: 1) 10s dump of the process state during copy shows: - Ext4: 526 probes done, 34 hits R state, 492 hits D state - Btrfs (2.6.39.4): 218, 83, 135 - Btrfs (3.2.7): 238, 62, 174, 2 hit sleeping 2) dd write/read of 55GB file to/from volume: - Ext4: write 127MB/s, read 107MB/s - Btrfs: 110MB/s, read 176MB/s -Jacek -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
