I think it's caused by the kernel readahead, not glibc readahead.
TL;DR: This solves the problem:
echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb
Question: how to configure NFS/kernel to automatically set that?
Long version:
Doing step (4) below results in tremendous speedup:
1) mount -t nfs -o tcp,timeo=600,rsize=1048576,wsize=1048576
10.161.254.11:/srv/ltsp /mnt
2) cat /proc/fs/nfsfs/volumes
We see the DEV number from there, e.g. 0:58
3) cat /sys/devices/virtual/bdi/0:58/read_ahead_kb
15360
I assume that this means the kernel will try to read ahead up to 15 MB
for each accessed file. *THIS IS THE PROBLEM*. For non-NFS devices, this
value is 128 (KB).
4) echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb
5) Test. Traffic now should be a *lot* less, and speed a *lot* more.
E.g. my NFS booting tests:
- read_ahead_kb=15360 (the default) => 1160 MB traffic to boot
- read_ahead_kb=128 => 324MB traffic
- read_ahead_kb=4 => 223MB traffic
So the question that remains, is how to properly configure either NFS or
the kernel, to use small readahead values for NFS.
I'm currently doing it with this workaround:
for f in $(awk '/^v[0-9]/ { print $4 }' < /proc/fs/nfsfs/volumes); do
echo 4 > /sys/devices/virtual/bdi/$f/read_ahead_kb; done
Thanks,
Alkis
