Re: [PATCH v6 00/11] VFS hot tracking

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Ping ^ 7

On Wed, Nov 6, 2013 at 9:45 PM, Zhi Yong Wu <zwu.kernel@xxxxxxxxx> wrote:
> From: Zhi Yong Wu <wuzhy@xxxxxxxxxxxxxxxxxx>
>
>   The patchset is trying to introduce hot tracking function in
> VFS layer, which will keep track of real disk I/O in memory.
> By it, you will easily know more details about disk I/O, and
> then detect where disk I/O hot spots are. Also, specific FS
> can take use of it to do accurate defragment, and hot relocation
> support, etc.
>
>   Now it's time to send out its V6 for external review, and
> any comments or ideas are appreciated, thanks.
>
> NOTE:
>
>   The patchset can be obtained via my kernel dev git on github:
> git://github.com/wuzhy/kernel.git hot_tracking
>   If you're interested, you can also review them via
> https://github.com/wuzhy/kernel/commits/hot_tracking
>
>   For how to use and more other info and performance report,
> please check hot_tracking.txt in Documentation and following
> links:
>   1.) http://lwn.net/Articles/525651/
>   2.) https://lkml.org/lkml/2012/12/20/199
>
>   This patchset has been done scalability or performance tests
> by fs_mark, ffsb and compilebench.
>
>   The perf testings were done on Linux 3.12.0-rc7 with Model IBM,8231-E2C
> Big Endian PPC64 with 64 CPUs and 2 NUMA nodes, 250G RAM and 1.50 TiB
> test hard disk where each test file size is 20G or 100G.
> Architecture:          ppc64
> Byte Order:            Big Endian
> CPU(s):                64
> On-line CPU(s) list:   0-63
> Thread(s) per core:    4
> Core(s) per socket:    1
> Socket(s):             16
> NUMA node(s):          2
> Model:                 IBM,8231-E2C
> Hypervisor vendor:     pHyp
> Virtualization type:   full
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              4096K
> NUMA node0 CPU(s):     0-31
> NUMA node1 CPU(s):     32-63
>
>   Below is the perf testing report:
>
>   Please focus on the two key points:
>   - The overall overhead which is injected by the patchset
>   - The stability of the perf results
>
> 1. fio tests
>
>                             w/o hot tracking                               w/ hot tracking
>
> RAM size                            32G          32G         16G           8G           4G           2G          250G
>
> sequential-8k-1jobs-read         61260KB/s    60918KB/s    60901KB/s    62610KB/s    60992KB/s    60213KB/s    60948KB/s
>
> sequential-8k-1jobs-write         1329KB/s     1329KB/s     1328KB/s     1329KB/s     1328KB/s     1329KB/s     1329KB/s
>
> sequential-8k-8jobs-read         91139KB/s    92614KB/s    90907KB/s    89895KB/s    92022KB/s    90851KB/s    91877KB/s
>
> sequential-8k-8jobs-write         2523KB/s     2522KB/s     2516KB/s     2521KB/s     2516KB/s     2518KB/s     2521KB/s
>
> sequential-256k-1jobs-read      151432KB/s   151403KB/s   151406KB/s   151422KB/s   151344KB/s   151446KB/s   151372KB/s
>
> sequential-256k-1jobs-write      33451KB/s    33470KB/s    33481KB/s    33470KB/s    33459KB/s    33472KB/s    33477KB/s
>
> sequential-256k-8jobs-read      235291KB/s   234555KB/s   234251KB/s   233656KB/s   234927KB/s   236380KB/s   235535KB/s
>
> sequential-256k-8jobs-write      62419KB/s    62402KB/s    62191KB/s    62859KB/s    62629KB/s    62720KB/s    62523KB/s
>
> random-io-mix-8k-1jobs  [READ]    2929KB/s     2942KB/s     2946KB/s     2929KB/s     2934KB/s     2947KB/s     2946KB/s
>                         [WRITE]   1262KB/s     1266KB/s     1257KB/s     1262KB/s     1257KB/s     1257KB/s     1265KB/s
>
> random-io-mix-8k-8jobs  [READ]    2444KB/s     2442KB/s     2436KB/s     2416KB/s     2353KB/s     2441KB/s     2442KB/s
>                         [WRITE]   1047KB/s     1044KB/s     1047KB/s     1028KB/s     1017KB/s     1034KB/s     1049KB/s
>
> random-io-mix-8k-16jobs [READ]    2182KB/s     2184KB/s     2169KB/s     2178KB/s     2190KB/s     2184KB/s     2180KB/s
>                         [WRITE]    932KB/s      930KB/s      943KB/s      936KB/s      937KB/s      929KB/s      931KB/s
>
> The above perf parameter is the aggregate bandwidth of threads in the group;
> If you hope to know how about other perf parameters, or fio raw results, please let me know, thanks.
>
> 2. Locking stat - Contention & Cacheline Bouncing
>
> RAM size         class name         con-bounces  contentions  acq-bounces   acquisitions   cacheline bouncing  locking contention
>                                                                                                  ratio              ratio
>
>               &(&root->t_lock)->rlock:  1508        1592         157834      374639292           0.96%              0.00%
> 250G          &(&root->m_lock)->rlock:  1469        1484         119221       43077842           1.23%              0.00%
>               &(&he->i_lock)->rlock:       0           0         101879      376755218           0.00%              0.00%
>
>               &(&root->t_lock)->rlock:  2912        2985         342575      374691186           0.85%              0.00%
> 32G           &(&root->m_lock)->rlock:   188         193         307765        8803163           0.00%              0.00%
>               &(&he->i_lock)->rlock:       0           0         291860      376756084           0.00%              0.00%
>
>               &(&root->t_lock)->rlock:  3863        3948         298041      374727038           1.30%              0.00%
> 16G           &(&root->m_lock)->rlock:   220         228         254451        8687057           0.00%              0.00%
>               &(&he->i_lock)->rlock:       0           0         235027      376756830           0.00%              0.00%
>
>               &(&root->t_lock)->rlock:  3283        3409         233790      374722064           1.40%              0.00%
> 8G            &(&root->m_lock)->rlock:   136         139         203917        8684313           0.00%              0.00%
>               &(&he->i_lock)->rlock:       0           0         193746      376756438           0.00%              0.00%
>
>               &(&root->t_lock)->rlock: 15090       15705         283460      374889666           5.32%              0.00%
> 4G            &(&root->m_lock)->rlock:   172         173         222480        8555052           0.00%              0.00%
>               &(&he->i_lock)->rlock:       0           0         206431      376759452           0.00%              0.00%
>
>               &(&root->t_lock)->rlock: 25515       27368         305129       375394828          8.36%              0.00%
> 2G            &(&root->m_lock)->rlock:   100         101         216516        6752265           0.00%              0.00%
>               &(&he->i_lock)->rlock:       0           0         214713      376765169           0.00%              0.00%
>
> 3. Perf test - Cacheline Ping-pong
>
>                       w/o hot tracking                                                        w/ hot tracking
>
> RAM size                    32G                  32G                 16G                  8G                   4G                    2G                  250G
>
> cache-references    1,264,996,437,581    1,401,504,955,577    1,398,308,614,801    1,396,525,544,527    1,384,793,467,410    1,432,042,560,409    1,571,627,148,771
>
> cache-misses           45,424,567,057       58,432,749,807       59,200,504,032       59,762,030,933       58,104,156,576       57,283,962,840       61,963,839,419
>
> seconds time elapsed  22956.327674298      23035.457069488      23017.232397085      23012.397142967      23008.420970731      23057.245578767      23342.456015188
>
> cache-misses ratio            3.591 %              4.169 %              4.234 %              4.279 %              4.196 %              4.000 %              3.943 %
>
> Changelog from v5:
>  - Also added the hook hot_freqs_update() in the page cache I/O path,
>    not only in real disk I/O path [viro]
>  - Don't export the stuff until it's used by a module [viro]
>  - Splitted hot_inode_item_lookup() [viro]
>  - Prevented hot items from being re-created after the inode was unlinked. [viro]
>  - Made hot_freqs_update() to be inline and adopt one private hot flag [viro]
>  - Killed hot_bit_shift() [viro]
>  - Used file_inode() instead of file->f_dentry->d_inode [viro]
>  - Introduced one new file hot_tracking.h in include/uapi/linux/ [viro]
>  - Made the checks for ->i_nlink to be protectd by ->i_mutex [viro]
>
> v5:
>  - Added all kinds of perf testing report [viro]
>  - Covered mmap() now [viro]
>  - Removed list_sort() in hot_update_worker() to avoid locking contention
>    and cacheline bouncing [viro]
>  - Removed a /proc interface to control low memory usage [Chandra]
>  - Adjusted shrinker support due to the change of public shrinker APIs [zwu]
>  - Fixed the locking missing issue when hot_inode_item_put() is called
>    in ioctl_heat_info() [viro]
>  - Fixed some locking contention issues [zwu]
>
> v4:
>  - Removed debugfs support, but leave it to TODO list [viro, Chandra]
>  - Killed HOT_DELETING and HOT_IN_LIST flag [viro]
>  - Fixed unlink issues [viro]
>  - Fixed the issue on lookups (both for inode and range)
>    leak on race with unlink  [viro]
>  - Killed hot_comm_item and split the functions which take it [virio]
>  - Fixed some other issues [zwu, Chandra]
>
> v3:
>  - Added memory caping function for hot items [Zhiyong]
>  - Cleanup aging function [Zhiyong]
>
> v2:
>  - Refactored to be under RCU [Chandra Seetharaman]
>   Merged some code changes [Chandra Seetharaman]
>  - Fixed some issues [Chandra Seetharaman]
>
> v1:
>  - Solved 64 bits inode number issue. [David Sterba]
>  - Embed struct hot_type in struct file_system_type [Darrick J. Wong]
>  - Cleanup Some issues [David Sterba]
>  - Use a static hot debugfs root [Greg KH]
>
> rfcv4:
>  - Introduce hot func registering framework [Zhiyong]
>  - Remove global variable for hot tracking [Zhiyong]
>  - Add btrfs hot tracking support [Zhiyong]
>
> rfcv3:
>  1.) Rewritten debugfs support based seq_file operation. [Dave Chinner]
>  2.) Refactored workqueue support. [Dave Chinner]
>  3.) Turn some Micro into be tunable [Zhiyong, Liu Zheng]
>      TIME_TO_KICK, and HEAT_UPDATE_DELAY
>  4.) Cleanedup a lot of other issues [Dave Chinner]
>
>
> rfcv2:
>  1.) Converted to Radix trees, not RB-tree [Zhiyong, Dave Chinner]
>  2.) Added memory shrinker [Dave Chinner]
>  3.) Converted to one workqueue to update map info periodically [Dave Chinner]
>  4.) Cleanedup a lot of other issues [Dave Chinner]
>
> rfcv1:
>  1.) Reduce new files and put all in fs/hot_tracking.[ch] [Dave Chinner]
>  2.) The first three patches can probably just be flattened into one.
>                                         [Marco Stornelli , Dave Chinner]
>
>
> Dave Chinner (1):
>   VFS hot tracking, xfs: Add hot tracking support
>
> Zhi Yong Wu (10):
>   VFS hot tracking: Define basic data structures and functions
>   VFS hot tracking: Track IO and record heat information
>   VFS hot tracking: Add a workqueue to move items between hot maps
>   VFS hot tracking: Add shrinker functionality to curtail memory usage
>   VFS hot tracking: Add an ioctl to get hot tracking information
>   VFS hot tracking: Add a /proc interface to make the interval tunable
>   VFS hot tracking: Add a /proc interface to control memory usage
>   VFS hot tracking: Add documentation
>   VFS hot tracking, btrfs: Add hot tracking support
>   MAINTAINERS: add the maintainers for VFS hot tracking
>
>  Documentation/filesystems/00-INDEX         |   2 +
>  Documentation/filesystems/hot_tracking.txt | 207 ++++++++
>  MAINTAINERS                                |  12 +
>  fs/Makefile                                |   2 +-
>  fs/btrfs/ctree.h                           |   1 +
>  fs/btrfs/super.c                           |  22 +-
>  fs/compat_ioctl.c                          |   5 +
>  fs/dcache.c                                |   2 +
>  fs/hot_tracking.c                          | 816 +++++++++++++++++++++++++++++
>  fs/hot_tracking.h                          |  72 +++
>  fs/ioctl.c                                 |  71 +++
>  fs/namei.c                                 |   4 +
>  fs/xfs/xfs_mount.h                         |   1 +
>  fs/xfs/xfs_super.c                         |  18 +
>  include/linux/fs.h                         |   4 +
>  include/linux/hot_tracking.h               | 107 ++++
>  include/uapi/linux/fs.h                    |   1 +
>  include/uapi/linux/hot_tracking.h          |  33 ++
>  kernel/sysctl.c                            |  14 +
>  mm/filemap.c                               |  24 +-
>  mm/readahead.c                             |   6 +
>  21 files changed, 1420 insertions(+), 4 deletions(-)
>  create mode 100644 Documentation/filesystems/hot_tracking.txt
>  create mode 100644 fs/hot_tracking.c
>  create mode 100644 fs/hot_tracking.h
>  create mode 100644 include/linux/hot_tracking.h
>  create mode 100644 include/uapi/linux/hot_tracking.h
>
> --
> 1.7.11.7
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
Regards,

Zhi Yong Wu
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux