[PATCH 00/11] [RFC] 512K readahead size with thrashing safe readahead

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Andrew,

This is to lift default readahead size to 512KB, which I believe yields
more I/O throughput without noticeably increasing I/O latency for today's HDD.

For example, for a 100MB/s and 8ms access time HDD:

io_size KB      access_time   transfer_time  io_latency  util%    throughput KB/s IOPS
4               8             0.04           8.04        0.49%    497.57          124.39
8               8             0.08           8.08        0.97%    990.33          123.79
16              8             0.16           8.16        1.92%    1961.69         122.61
32              8             0.31           8.31        3.76%    3849.62         120.30
64              8             0.62           8.62        7.25%    7420.29         115.94
128             8             1.25           9.25        13.51%   13837.84        108.11
256             8             2.50           10.50       23.81%   24380.95        95.24
512             8             5.00           13.00       38.46%   39384.62        76.92
1024            8             10.00          18.00       55.56%   56888.89        55.56
2048            8             20.00          28.00       71.43%   73142.86        35.71
4096            8             40.00          48.00       83.33%   85333.33        20.83

The 128KB => 512KB readahead size boosts IO throughput from ~13MB/s to ~39MB/s, while
merely increases IO latency from 9.25ms to 13.00ms.

As for SSD, I find that Intel X25-M SSD desires large readahead size
even for sequential reads (the first patch has benchmark details):

        rasize  first run time/throughput       second run time/throughput
        ------------------------------------------------------------------
          4k    3.40038 s,      123 MB/s        3.42842 s,      122 MB/s
          8k    2.7362 s,       153 MB/s        2.74528 s,      153 MB/s
         16k    2.59808 s,      161 MB/s        2.58728 s,      162 MB/s
         32k    2.50488 s,      167 MB/s        2.49138 s,      168 MB/s
         64k    2.12861 s,      197 MB/s        2.13055 s,      197 MB/s
        128k    1.92905 s,      217 MB/s        1.93176 s,      217 MB/s
        256k    1.75896 s,      238 MB/s        1.78963 s,      234 MB/s
        512k    1.67357 s,      251 MB/s        1.69112 s,      248 MB/s
          1M    1.62115 s,      259 MB/s        1.63206 s,      257 MB/s
          2M    1.56204 s,      269 MB/s        1.58854 s,      264 MB/s
          4M    1.57949 s,      266 MB/s        1.57426 s,      266 MB/s

As suggested by Linus, decrease default readahead size for small devices at the same time.

	[PATCH 01/11] readahead: limit readahead size for small devices
	[PATCH 02/11] readahead: bump up the default readahead size
	[PATCH 03/11] readahead: introduce {MAX|MIN}_READAHEAD_PAGES macros for ease of use

The two other impacts of an enlarged readahead size are

- memory footprint (caused by readahead miss)
	Sequential readahead hit ratio is pretty high regardless of max
	readahead size; the extra memory footprint is mainly caused by
	enlarged mmap read-around.
	I measured my desktop:
	- under Xwindow:
		128KB readahead cache hit ratio = 143MB/230MB = 62%
		512KB readahead cache hit ratio = 138MB/248MB = 55%
	- under console: (seems more stable than the Xwindow data)
		128KB readahead cache hit ratio = 30MB/56MB   = 53%
		  1MB readahead cache hit ratio = 30MB/59MB   = 51%
	So the impact to memory footprint looks acceptable.

- readahead thrashing
	It will now cost 1MB readahead buffer per stream.  Memory tight systems
	typically do not run multiple streams; but if they do so, it should
	help I/O performance as long as we can avoid thrashing, which can be
	achieved with the following patches.

	[PATCH 04/11] readahead: replace ra->mmap_miss with ra->ra_flags
	[PATCH 05/11] readahead: retain inactive lru pages to be accessed soon
	[PATCH 06/11] readahead: thrashing safe context readahead

This is a major rewrite of the readahead algorithm, so I did careful tests with
the following tracing/stats patches:

	[PATCH 07/11] readahead: record readahead patterns
	[PATCH 08/11] readahead: add tracing event
	[PATCH 09/11] readahead: add /debug/readahead/stats

I verified the new readahead behavior on various access patterns,
as well as stress tested the thrashing safety, by running 300 streams
with mem=128M.

Only 2031/61325=3.3% readahead windows are thrashed (due to workload
variation):

# cat /debug/readahead/stats
pattern     readahead    eof_hit  cache_hit         io    sync_io    mmap_io       size async_size    io_size
initial            20          9          4         20         20         12         73         37         35
subsequent          3          3          0          1          0          1          8          8          1
context         61325          1       5479      61325       6788          5         14          2         13
thrash           2031          0       1222       2031       2031          0          9          0          6
around            235         90        142        235        235        235         60          0         19
fadvise             0          0          0          0          0          0          0          0          0
random            223        133          0         91         91          1          1          0          1
all             63837        236       6847      63703       9165          0         14          2         13

And the readahead inside a single stream is working as expected:

# grep streams-3162 /debug/tracing/trace
         streams-3162  [000]  8602.455953: readahead: readahead-context(dev=0:2, ino=0, req=287352+1, ra=287354+10-2, async=1) = 10
         streams-3162  [000]  8602.907873: readahead: readahead-context(dev=0:2, ino=0, req=287362+1, ra=287364+20-3, async=1) = 20
         streams-3162  [000]  8604.027879: readahead: readahead-context(dev=0:2, ino=0, req=287381+1, ra=287384+14-2, async=1) = 14
         streams-3162  [000]  8604.754722: readahead: readahead-context(dev=0:2, ino=0, req=287396+1, ra=287398+10-2, async=1) = 10
         streams-3162  [000]  8605.191228: readahead: readahead-context(dev=0:2, ino=0, req=287406+1, ra=287408+18-3, async=1) = 18
         streams-3162  [000]  8606.831895: readahead: readahead-context(dev=0:2, ino=0, req=287423+1, ra=287426+12-2, async=1) = 12
         streams-3162  [000]  8606.919614: readahead: readahead-thrash(dev=0:2, ino=0, req=287425+1, ra=287425+8-0, async=0) = 1
         streams-3162  [000]  8607.545016: readahead: readahead-context(dev=0:2, ino=0, req=287436+1, ra=287438+9-2, async=1) = 9
         streams-3162  [000]  8607.960039: readahead: readahead-context(dev=0:2, ino=0, req=287445+1, ra=287447+18-3, async=1) = 18
         streams-3162  [000]  8608.790973: readahead: readahead-context(dev=0:2, ino=0, req=287462+1, ra=287465+21-3, async=1) = 21
         streams-3162  [000]  8609.763138: readahead: readahead-context(dev=0:2, ino=0, req=287483+1, ra=287486+15-2, async=1) = 15
         streams-3162  [000]  8611.467401: readahead: readahead-context(dev=0:2, ino=0, req=287499+1, ra=287501+11-2, async=1) = 11
         streams-3162  [000]  8642.512413: readahead: readahead-context(dev=0:2, ino=0, req=288053+1, ra=288056+10-2, async=1) = 10
         streams-3162  [000]  8643.246618: readahead: readahead-context(dev=0:2, ino=0, req=288064+1, ra=288066+22-3, async=1) = 22
         streams-3162  [000]  8644.278613: readahead: readahead-context(dev=0:2, ino=0, req=288085+1, ra=288088+16-3, async=1) = 16
         streams-3162  [000]  8644.395782: readahead: readahead-context(dev=0:2, ino=0, req=288087+1, ra=288087+21-3, async=0) = 5
         streams-3162  [000]  8645.109918: readahead: readahead-context(dev=0:2, ino=0, req=288101+1, ra=288108+8-1, async=1) = 8
         streams-3162  [000]  8645.285078: readahead: readahead-context(dev=0:2, ino=0, req=288105+1, ra=288116+8-1, async=1) = 8
         streams-3162  [000]  8645.731794: readahead: readahead-context(dev=0:2, ino=0, req=288115+1, ra=288122+14-2, async=1) = 13
         streams-3162  [000]  8646.114250: readahead: readahead-context(dev=0:2, ino=0, req=288123+1, ra=288136+8-1, async=1) = 8
         streams-3162  [000]  8646.626320: readahead: readahead-context(dev=0:2, ino=0, req=288134+1, ra=288144+16-3, async=1) = 16
         streams-3162  [000]  8647.035721: readahead: readahead-context(dev=0:2, ino=0, req=288143+1, ra=288160+10-2, async=1) = 10
         streams-3162  [000]  8647.693082: readahead: readahead-context(dev=0:2, ino=0, req=288157+1, ra=288165+12-2, async=1) = 8
         streams-3162  [000]  8648.221368: readahead: readahead-context(dev=0:2, ino=0, req=288168+1, ra=288177+15-2, async=1) = 15
         streams-3162  [000]  8649.280800: readahead: readahead-context(dev=0:2, ino=0, req=288190+1, ra=288192+23-3, async=1) = 23
	 [...]

btw, Linus suggested to disable start-of-file readahead if lseek() has been called:

	[PATCH 10/11] readahead: dont do start-of-file readahead after lseek()

At last, the updated context readahead will do more radix tree scans, so need
to optimize radix_tree_prev_hole():

	[PATCH 11/11] radixtree: speed up next/prev hole search

It will on average reduce 8*64 level-0 slot searches to 32 level-0 slot
plus 8 level-1 node searches.

Thanks,
Fengguang

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux