On Mon, Jul 18, 2011 at 05:31:09PM -0400, Keiichi KII wrote: > Hello, > > I would propose pagecache monitoring tools using perf tools. > The purpose of the tools is to clarify pagecache behavior in a system. > > We can now know system-wide pagecache usage by "/proc/meminfo". > But we don't have any way to get higher resolution information like > per file or per process usage than system-wide one. It's convulated and slow but it can be determined what processes are mapping pages from a file by scanning /proc/PID/maps to identify what files are mapped and then using /proc/PID/pageflags to see what pages are mapped on a per-process basis. > If pagecache miss hit ratio rises due to unnecessary adding/removing > pagecaches, maybe it leads to extra I/O and affects system performance. > But it's difficult to find out what is going on in the system. > What is "unnecessary adding/removing pagecaches"? If it's added, it means either someone has faulted the page or it is due to readahead. It's necessary to something. > So, the tools I propose provide 2 functions: > > 1. pagecache snapshooting(perf script pagecache-snapshoot) > snapshot, not snapshoot :) > This function clarifies pagecache usage per each file in the system. > This function is based mainly on "pagecache object collections" that is > developed by Wu Fengguang (http://lkml.org/lkml/2010/2/9/156). > The following is sample output of this function. > > pagecache snapshooting (time: 14131, path: /home) > file name cache(B) file(B) ratio +/-(B) age > -------------------------------------- -------- -------- ------ ------- ------ > /home/foo/git/linux-2.6-tip/.git/objec 71.0M 436.6M 16% 0 9012 > /home/foo/git/linux-2.6-tip/.git/objec 49.6M 57.7M 86% 0 9012 > /home/foo/.thunderbird/xso5zn7g.defaul 19.8M 19.8M 100% 0 7223 > /home/foo/.thunderbird/xso5zn7g.defaul 5.7M 5.7M 100% 0 6621 > /home/foo/git/linux-2.6-tip/.git/index 3.5M 3.5M 100% 0 4306 > /home/foo/.thunderbird/xso5zn7g.defaul 2.2M 2.2M 100% 0 7524 > /home/foo/.thunderbird/xso5zn7g.defaul 2.2M 2.2M 100% 0 7526 > /home/foo/.thunderbird/xso5zn7g.defaul 1.7M 1.7M 100% 0 6921 > ... > > 2. continuous pagecache monitoring(perf script pagecachetop) > > This function clarifies pagecache behavior like pagecache hit ratio and > added/removed pagecache amount on the basis of file/process. > This functions is based on pagecache tracepoints I propose. > While the pagecache snapshooting can take a pagecache snapshoot at a point, > the continuous pagecache monitoring can measure dynamic change between > 2 snapshoots. > The following is sample output of this function. > > pagecache behavior per file (time:15826, interval:10) > > find hit cache add remove proc > file count ratio pages(B) pages(B) pages(B) count > -------------------- -------- ---------- -------- -------- -------- ----- > libc-2.13.so 620 100.00% 1.2M 0 0 7 > bash 283 100.00% 888.0K 0 0 6 > ld-2.13.so 136 100.00% 148.0K 0 0 6 > gawk 130 100.00% 376.0K 0 0 2 > ld.so.cache 60 100.00% 116.0K 0 0 4 > ... > > pagecache behavior per process (time:16294, interval:10) > > find hit add remove file > process count ratio pages(B) pages(B) count > -------------------- -------- ---------- -------- -------- ----- > zsh-7761 2968 99.93% 4.0K 0 246 > perf-7758 369 100.00% 0 0 17 > xmms-7634 52 100.00% 0 0 1 > perf-7759 11 100.00% 0 0 2 > zsh-2815 6 83.33% 4.0K 4.0K 2 > gconfd-2-4849 3 0.00% 12.0K 12.0K 4 > rsyslogd-7194 1 100.00% 0 0 1 > Ok, I can see how this could identify hot files in the system and processes that are currently active but I'm not seeing what problem could be solved with this information. For example, a process that is suffering a poor hit ratio can usually be identified with a combination of vmstat showing a large amount of activity with page in/out or swap in/out and a profile showing a process that is spending a high percentage of its time servicing faults. If a profile is unsuitable, picking active processes from top and then monitoring /proc/PID/stat and the fault counters can identify which process in trouble. Can you give an example of a problem that using this tool would identify and help resolve? > By these 2 functions, we can find out whether pagecaches are used > efficiently or not. > And also these tools would help us tune some applications like database. If the database is using direct IO, will it even show up in this tool? > It will also help us tune the kernel parameters like "vm.dirty_*". > Does it though? A dirty_ratio that is too low can cause stalls as processes get throttled cleaning pages but the pages are still present in the page cache and the hit ratio would not be affected. I'm not saying this is a bad idea but the changelog needs to be improved to explain why this information is helpful and why the existing mechanisms are either unsuitable or just too awkward to use. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>