On 01/02/18 01:34, Jeremy Linton wrote:
On 01/31/2018 09:49 AM, J. Bruce Fields wrote:
On Tue, Jan 30, 2018 at 01:52:49PM -0600, Jeremy Linton wrote:
Have you tried this with a '-o nfsvers=3' during mount? Did that help?
I noticed a large decrease in my kernel build times across NFS/lan a
while
back after a machine/kernel/10g upgrade. After playing with
mount/export
options filesystem tuning/etc, I got to this point of timing a bunch of
these operations vs the older machine, at which point I discovered that
simply backing down to NFSv3 solved the problem.
AKA a nfsv3 server on a 10 year old 4 disk xfs RAID5 on 1Gb
ethernet, was
slower than a modern machine with a 8 disk xfs RAID5 on 10Gb on
nfsv4. The
effect was enough to change a kernel build from ~45 minutes down to
less
than 5.
Using NFSv3 in async mode is faster than NFSv4 in async mode (still
abysmal in sync mode).
NFSv3 async: sync; time (tar -xf linux-4.14.15.tar.gz -C /data2/tmp; sync)
real 2m25.717s
user 0m8.739s
sys 0m13.362s
NFSv4 async: sync; time (tar -xf linux-4.14.15.tar.gz -C /data2/tmp; sync)
real 3m33.032s
user 0m8.506s
sys 0m16.930s
NFSv3 async: wireshark trace
No. Time Source Destination Protocol Length Info
18527 2.815884979 192.168.202.2 192.168.202.1 NFS
216 V3 CREATE Call (Reply In 18528), DH: 0x62f39428/dma.h Mode: EXCLUSIVE
18528 2.816362338 192.168.202.1 192.168.202.2 NFS
328 V3 CREATE Reply (Call In 18527)
18529 2.816418841 192.168.202.2 192.168.202.1 NFS
224 V3 SETATTR Call (Reply In 18530), FH: 0x13678ba0
18530 2.816871820 192.168.202.1 192.168.202.2 NFS
216 V3 SETATTR Reply (Call In 18529)
18531 2.816966771 192.168.202.2 192.168.202.1 NFS
1148 V3 WRITE Call (Reply In 18532), FH: 0x13678ba0 Offset: 0 Len: 934
FILE_SYNC
18532 2.817441291 192.168.202.1 192.168.202.2 NFS
208 V3 WRITE Reply (Call In 18531) Len: 934 FILE_SYNC
18533 2.817495775 192.168.202.2 192.168.202.1 NFS
236 V3 SETATTR Call (Reply In 18534), FH: 0x13678ba0
18534 2.817920346 192.168.202.1 192.168.202.2 NFS
216 V3 SETATTR Reply (Call In 18533)
18535 2.818002910 192.168.202.2 192.168.202.1 NFS
216 V3 CREATE Call (Reply In 18536), DH: 0x62f39428/elf.h Mode: EXCLUSIVE
18536 2.818492126 192.168.202.1 192.168.202.2 NFS
328 V3 CREATE Reply (Call In 18535)
This is taking about 2ms for a small file write rather than 3ms for
NFSv4. There is an extra GETATTR and CLOSE RPC in NFSv4 accounting for
the difference.
So where I am:
1. NFS in sync mode, at least on my two Fedora27 systems for my usage is
completely unusable. (sync: 2 hours, async: 3 minutes, localdisk: 13
seconds).
2. NFS async mode is working, but the small writes are still very slow.
3. NFS in async mode is 30% better with NFSv3 than NFSv4 when writing
small files due to the increased latency caused by NFSv4's two extra RPC
calls.
I really think that in 2018 we should be able to have better NFS
performance when writing many small files such as used in software
development. This would speed up any system that was using NFS with this
sort of workload dramatically and reduce power usage all for some
improvements in the NFS protocol.
I don't know the details of if this would work, or who is responsible
for NFS, but it would be good if possible to have some improvements
(NFSv4.3 ?). Maybe:
1. Have an OPEN-SETATTR-WRITE RPC call all in one and a SETATTR-CLOSE
call all in one. This would reduce the latency of a small file to 1ms
rather than 3ms thus 66% faster. Would require the client to delay the
OPEN/SETATTR until the first WRITE. Not sure how possible this is in the
implementations. Maybe READ's could be improved as well but getting the
OPEN through quick may be better in this case ?
2. Could go further with an OPEN-SETATTR-WRITE-CLOSE RPC call. (0.5ms vs
3ms).
3. On sync/async modes personally I think it would be better for the
client to request the mount in sync/async mode. The setting of sync on
the server side would just enforce sync mode for all clients. If the
server is in the default async mode clients can mount using sync or
async as to their requirements. This seems to match normal VFS semantics
and usage patterns better.
4. The 0.5ms RPC latency seems a bit high (ICMP pings 0.12ms) . Maybe
this is worth investigating in the Linux kernel processing (how ?) ?
5. The 20ms RPC latency I see in sync mode needs a look at on my system
although async mode is fine for my usage. Maybe this ends up as 2 x 10ms
drive seeks on ext4 and is thus expected.
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
