On 9/21/19 10:52 AM, Alkis Georgopoulos wrote:
I think it's caused by the kernel readahead, not glibc readahead.
TL;DR: This solves the problem:
echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb
Question: how to configure NFS/kernel to automatically set that?
Long version:
Doing step (4) below results in tremendous speedup:
1) mount -t nfs -o tcp,timeo=600,rsize=1048576,wsize=1048576
10.161.254.11:/srv/ltsp /mnt
2) cat /proc/fs/nfsfs/volumes
We see the DEV number from there, e.g. 0:58
3) cat /sys/devices/virtual/bdi/0:58/read_ahead_kb
15360
I assume that this means the kernel will try to read ahead up to 15 MB
for each accessed file. *THIS IS THE PROBLEM*. For non-NFS devices, this
value is 128 (KB).
4) echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb
5) Test. Traffic now should be a *lot* less, and speed a *lot* more.
E.g. my NFS booting tests:
- read_ahead_kb=15360 (the default) => 1160 MB traffic to boot
- read_ahead_kb=128 => 324MB traffic
- read_ahead_kb=4 => 223MB traffic
So the question that remains, is how to properly configure either NFS or
the kernel, to use small readahead values for NFS.
I'm currently doing it with this workaround:
for f in $(awk '/^v[0-9]/ { print $4 }' < /proc/fs/nfsfs/volumes); do
echo 4 > /sys/devices/virtual/bdi/$f/read_ahead_kb; done
Thanks,
Alkis
Quoting https://lkml.org/lkml/2010/2/26/48
> nfs: use 2*rsize readahead size
> With default rsize=512k and NFS_MAX_READAHEAD=15, the current NFS
> readahead size 512k*15=7680k is too large than necessary for typical
> clients.
I.e. the problem probably is that when NFS_MAX_READAHEAD=15 was
implemented, rsize was 512k; now that rsize=1M, this results in
readaheads of 15M, which cause all the traffic and lags.
