Hello Dave Chinner, ________________________________________ From: Dave Chinner [david@xxxxxxxxxxxxx] Sent: Tuesday, July 05, 2016 8:18 To: Wang Shilong Cc: linux-xfs@xxxxxxxxxxxxxxx; xfs@xxxxxxxxxxx Subject: Re: Bad Metadata performances for XFS? On Tue, Jul 05, 2016 at 08:52:26AM +1000, Dave Chinner wrote: > [xfs@xxxxxxxxxxx is where you'll find the XFS developers] > > On Mon, Jul 04, 2016 at 05:32:40AM +0000, Wang Shilong wrote: > > Hello Guys, > > > > I happened run some benchmarks for XFS, and found some intresting to share here: > > Kernel version: > > [root@localhost shm]# uname -r > > 4.7.0-rc5+ > > > > [root@localhost shm]# cat /proc/cpuinfo | grep Intel > > vendor_id : GenuineIntel > > model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz What's the rest of the hardware in the machine? [root@localhost ~]# cat /proc/meminfo MemTotal: 32823104 kB MemFree: 29981320 kB MemAvailable: 31672712 kB Buffers: 6176 kB Cached: 1241192 kB SwapCached: 0 kB Active: 938332 kB Inactive: 692420 kB Active(anon): 384576 kB Inactive(anon): 111320 kB Active(file): 553756 kB Inactive(file): 581100 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 11324 kB Writeback: 0 kB AnonPages: 383496 kB Mapped: 186516 kB Shmem: 112496 kB Slab: 1059544 kB SReclaimable: 1020764 kB SUnreclaim: 38780 kB KernelStack: 6896 kB PageTables: 22160 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 16411552 kB Committed_AS: 2382640 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 194868 kB DirectMap2M: 3874816 kB DirectMap1G: 30408704 kB [root@localhost ~]# dmidecode -t memory # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. Handle 0x0041, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: None Maximum Capacity: 32 GB Error Information Handle: Not Provided Number Of Devices: 4 Handle 0x0042, DMI type 17, 40 bytes Memory Device Array Handle: 0x0041 Error Information Handle: Not Provided Total Width: 64 bits Data Width: 64 bits Size: 8192 MB Form Factor: DIMM Set: None Locator: ChannelA-DIMM0 Bank Locator: BANK 0 Type: DDR3 Type Detail: Synchronous Speed: 1600 MHz Manufacturer: Kingston Serial Number: 692A784E Asset Tag: 9876543210 Part Number: KHX1600C10D3/8GX Rank: 2 Configured Clock Speed: 1333 MHz Minimum Voltage: 1.5 V Maximum Voltage: 1.5 V Configured Voltage: 1.5 V Handle 0x0044, DMI type 17, 40 bytes Memory Device Array Handle: 0x0041 Error Information Handle: Not Provided Total Width: 64 bits Data Width: 64 bits Size: 8192 MB Form Factor: DIMM Set: None Locator: ChannelA-DIMM1 Bank Locator: BANK 1 Type: DDR3 Type Detail: Synchronous Speed: 1600 MHz Manufacturer: Kingston Serial Number: 672A954E Asset Tag: 9876543210 Part Number: KHX1600C10D3/8GX Rank: 2 Configured Clock Speed: 1333 MHz Minimum Voltage: 1.5 V Maximum Voltage: 1.5 V Configured Voltage: 1.5 V Handle 0x0046, DMI type 17, 40 bytes Memory Device Array Handle: 0x0041 Error Information Handle: Not Provided Total Width: 64 bits Data Width: 64 bits Size: 8192 MB Form Factor: DIMM Set: None Locator: ChannelB-DIMM0 Bank Locator: BANK 2 Type: DDR3 Type Detail: Synchronous Speed: 1600 MHz Manufacturer: Kingston Serial Number: 712AE08D Asset Tag: 9876543210 Part Number: KHX1600C10D3/8GX Rank: 2 Configured Clock Speed: 1333 MHz Minimum Voltage: 1.5 V Maximum Voltage: 1.5 V Configured Voltage: 1.5 V Handle 0x0048, DMI type 17, 40 bytes Memory Device Array Handle: 0x0041 Error Information Handle: Not Provided Total Width: 64 bits Data Width: 64 bits Size: 8192 MB Form Factor: DIMM Set: None Locator: ChannelB-DIMM1 Bank Locator: BANK 3 Type: DDR3 Type Detail: Synchronous Speed: 1600 MHz Manufacturer: Kingston Serial Number: 6A2A144E Asset Tag: 9876543210 Part Number: KHX1600C10D3/8GX Rank: 2 Configured Clock Speed: 1333 MHz Minimum Voltage: 1.5 V Maximum Voltage: 1.5 V Configured Voltage: 1.5 V > > dd 16GB to /dev/shm/data to use memory backend storage to benchmark metadata performaces. I've never seen anyone create a ramdisk like that before. What's the backing device type? i.e. what block device driver does this use? I guess you mean loop device here? It is common file and setup as loop0 device here. > > Benchmark tool is mdtest, you can download it from > > https://sourceforge.net/projects/mdtest/ What version? The sourceforge version, of the github fork that the sourceforge page points to? Or the forked branch of recent development in the github fork? I don't think sourceforge version or github version make some differences here, you could use any of them.(I used Souceforge version) > > Steps to run benchmark > > #mkfs.xfs /dev/shm/data Output of this command so we can recreate the same filesystem structure? [root@localhost shm]# mkfs.xfs data meta-data=data isize=512 agcount=4, agsize=1025710 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0 data = bsize=4096 blocks=4102840, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 > > #mount /dev/shm/data /mnt/test > > #mdtest -d /mnt/test -n 2000000 > > > > 1 tasks, 2000000 files/directories > > > > SUMMARY: (of 1 iterations) > > Operation Max Min Mean Std Dev > > --------- --- --- ---- ------- > > Directory creation: 24724.717 24724.717 24724.717 0.000 > > Directory stat : 1156009.290 1156009.290 1156009.290 0.000 > > Directory removal : 103496.353 103496.353 103496.353 0.000 > > File creation : 23094.444 23094.444 23094.444 0.000 > > File stat : 1158704.969 1158704.969 1158704.969 0.000 > > File read : 752731.595 752731.595 752731.595 0.000 > > File removal : 105481.766 105481.766 105481.766 0.000 > > Tree creation : 2229.827 2229.827 2229.827 0.000 > > Tree removal : 1.275 1.275 1.275 0.000 > > > > -- finished at 07/04/2016 12:54:26 -- A table of numbers with no units or explanation as to what they mean. Let me guess - I have to read the benchmark source code to understand what the numbers mean? You could look File Creation, Units mean number of files create per seconds. (Here it is 23094.444) > > IOPS for file creation is only 2.3W, however compare to Ext4 with same testing. Ummm - what unit of measurement is "W"? Watts? Sorry, same as above.. Please, when presenting benchmark results to ask for help with analysis, be *extremely specific* about what you running and what the results mean. It's no different from reporting a bug from this perspective: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F That said, this is a single threaded benchmark. It's well known that XFS uses more CPU per metadata operation than either ext4 or btrfs, so it won't be any surprise that they are faster than XFS on this particular test. We've known this for many years now - perhaps you should watch/read this presentation I did more than 4 years ago now: http://xfs.org/index.php/File:Xfs-scalability-lca2012.pdf http://www.youtube.com/watch?v=FegjLbCnoBw IOWs: Being CPU bound at 25,000 file creates/s is in line with what I'd expect on XFS for a single threaded, single directory create over 2 million directory entries with the default 4k directory block size.... ---------- I understand that this is single thread Limit, but I guess there are some other Limit here, because even single thread creating 50W files speed is twice than 200W files. Thanks, Shilong Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
