On Wed, Feb 01, 2012 at 07:25:26AM +1100, Dave Chinner wrote: > The only thing changing the inode size will have affected is the > directory structure - maybe your directories are small enough to fit > in line, or the inode is large enough to keep it in extent format > rather than a full btree. In either case, though, the directory > lookup will require less IO. I've done a whole bunch of testing, which I won't describe in detail unless you're interested, but I've finally found out what's causing the sudden change in performance. With defaults, the files in one directory are spread all over the filesystem. But with -i size=1024, the files in a directory are stored adjacent to each other. Hence reading all the files in one directory requires far less seeking across the disk, and runs about 3 times faster. Here is the filesystem on a disk formatted with defaults: root@storage1:~# find /data/sdc | head -20 | xargs xfs_bmap /data/sdc: no extents /data/sdc/Bonnie.26384: 0: [0..31]: 567088..567119 /data/sdc/Bonnie.26384/00000: 0: [0..7]: 567120..567127 /data/sdc/Bonnie.26384/00000/0icoeTRPHKX0000000000: 0: [0..1015]: 4411196808..4411197823 /data/sdc/Bonnie.26384/00000/Q0000000001: 0: [0..1543]: 1466262056..1466263599 /data/sdc/Bonnie.26384/00000/JFXQyeq6diG0000000002: 0: [0..1295]: 2936342144..2936343439 /data/sdc/Bonnie.26384/00000/TK7ciXkkj0000000003: 0: [0..1519]: 4411197824..4411199343 /data/sdc/Bonnie.26384/00000/0000000004: 0: [0..1207]: 1466263600..1466264807 /data/sdc/Bonnie.26384/00000/acJKZWAwEnu0000000005: 0: [0..1223]: 2936343440..2936344663 /data/sdc/Bonnie.26384/00000/9wIgxPKeI4B0000000006: 0: [0..1319]: 4411199344..4411200663 /data/sdc/Bonnie.26384/00000/C6QLFdND0000000007: 0: [0..1111]: 1466264808..1466265919 /data/sdc/Bonnie.26384/00000/6xc1Wydh0000000008: 0: [0..1223]: 2936344664..2936345887 /data/sdc/Bonnie.26384/00000/0000000009: 0: [0..1167]: 4411200664..4411201831 /data/sdc/Bonnie.26384/00000/HdlN0000000000a: 0: [0..1535]: 1466265920..1466267455 /data/sdc/Bonnie.26384/00000/52IabyC5pvis000000000b: 0: [0..1287]: 2936345888..2936347175 /data/sdc/Bonnie.26384/00000/LvDhxcdLf000000000c: 0: [0..1583]: 4411201832..4411203415 /data/sdc/Bonnie.26384/00000/08P3JAR000000000d: 0: [0..1255]: 1466267456..1466268711 /data/sdc/Bonnie.26384/00000/000000000e: 0: [0..1095]: 2936347176..2936348271 /data/sdc/Bonnie.26384/00000/s0gtPGPecXu000000000f: 0: [0..1319]: 4411203416..4411204735 /data/sdc/Bonnie.26384/00000/HFLOcN0000000010: 0: [0..1503]: 1466268712..1466270215 And here is the filesystem created with -i size=1024: root@storage1:~# find /data/sdb | head -20 | xargs xfs_bmap /data/sdb: no extents /data/sdb/Bonnie.26384: 0: [0..7]: 243752..243759 1: [8..15]: 5526920..5526927 2: [16..23]: 7053272..7053279 3: [24..31]: 24223832..24223839 /data/sdb/Bonnie.26384/00000: 0: [0..7]: 1465133488..1465133495 /data/sdb/Bonnie.26384/00000/0icoeTRPHKX0000000000: 0: [0..1015]: 1465134032..1465135047 /data/sdb/Bonnie.26384/00000/Q0000000001: 0: [0..1543]: 1465135048..1465136591 /data/sdb/Bonnie.26384/00000/JFXQyeq6diG0000000002: 0: [0..1295]: 1465136592..1465137887 /data/sdb/Bonnie.26384/00000/TK7ciXkkj0000000003: 0: [0..1519]: 1465137888..1465139407 /data/sdb/Bonnie.26384/00000/0000000004: 0: [0..1207]: 1465139408..1465140615 /data/sdb/Bonnie.26384/00000/acJKZWAwEnu0000000005: 0: [0..1223]: 1465140616..1465141839 /data/sdb/Bonnie.26384/00000/9wIgxPKeI4B0000000006: 0: [0..1319]: 1465141840..1465143159 /data/sdb/Bonnie.26384/00000/C6QLFdND0000000007: 0: [0..1111]: 1465143160..1465144271 /data/sdb/Bonnie.26384/00000/6xc1Wydh0000000008: 0: [0..1223]: 1465144272..1465145495 /data/sdb/Bonnie.26384/00000/0000000009: 0: [0..1167]: 1465145496..1465146663 /data/sdb/Bonnie.26384/00000/HdlN0000000000a: 0: [0..1535]: 1465146664..1465148199 /data/sdb/Bonnie.26384/00000/52IabyC5pvis000000000b: 0: [0..1287]: 1465148200..1465149487 /data/sdb/Bonnie.26384/00000/LvDhxcdLf000000000c: 0: [0..1583]: 1465149488..1465151071 /data/sdb/Bonnie.26384/00000/08P3JAR000000000d: 0: [0..1255]: 1465151072..1465152327 /data/sdb/Bonnie.26384/00000/000000000e: 0: [0..1095]: 1465152464..1465153559 /data/sdb/Bonnie.26384/00000/s0gtPGPecXu000000000f: 0: [0..1319]: 1465153560..1465154879 /data/sdb/Bonnie.26384/00000/HFLOcN0000000010: 0: [0..1503]: 1465154880..1465156383 All the files in one directory are close to that directory; when you get to another directory the block offset jumps. This is a highly desirable property when you want to copy all the files: for example, using this filesystem I can tar it up and untar it onto another filesystem at 73MB/s, as compared to about 25MB/sec on a default filesystem. So now my questions now are: (1) Is this a fluke? What is it about -i size=1024 which causes this to happen? (2) What is the intended behaviour for XFS: that files should be close to their parent directory or spread across allocation groups? I did some additional tests: * -i size=512 Files spread around * -n size=16384 Files spread around * -i size=1024 -n size=16384 Files local to directory * -i size=2048 Files local to directory Any clues gratefully received. This usage pattern (dumping in a large library of files, and then processing all those files sequentially) is an important one for the system I'm working on. Regards, Brian. _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
