----- On 25 Nov, 2020, at 17:14, Daire Byrne daire@xxxxxxxx wrote: > First off, writing direct to the originating server mount on the re-export > server from userspace shows the ideal behaviour for all combinations: > > originating server <- (vers=X,actimeo=1800,nconnect=X) <- reexport server > writing = WRITE,WRITE .... repeating (good!) > > Then re-exporting a NFSv4.2 server: > > originating server <- (vers=4.2) <- reexport server - (vers=3) <- client writing > = GETATTR,COMMIT,WRITE .... repeating > originating server <- (vers=4.2) <- reexport server - (vers=4.2) <- client > writing = GETATTR,WRITE .... repeating > > And re-exporting a NFSv3 server: > > originating server <- (vers=3) <- reexport server - (vers=4.2) <- client writing > = WRITE,WRITE .... repeating (good!) > originating server <- (vers=3) <- reexport server - (vers=3) <- client writing = > WRITE,COMMIT .... repeating > > So of all the combinations, a NFSv4.2 re-export of an NFSv3 server is the only > one that matches the "ideal" case where we WRITE continuously without all the > extra chatter. > > And for completeness, taking that "good" case and making it bad with nconnect: > > originating server <- (vers=3,nconnect=16) <- reexport server - (vers=4.2) <- > client writing = WRITE,WRITE .... repeating (good!) > originating server <- (vers=3) <- reexport server <- (vers=4.2,nconnect=16) <- > client writing = WRITE,COMMIT,GETATTR .... randomly repeating > > So using nconnect on the re-export's client causes lots more metadata ops. There > are reasons for doing that for increasing throughput but it could be that the > gain is offset by the extra metadata roundtrips. > > Similarly, we have mostly been using a NFSv4.2 re-export of a NFSV4.2 server > over the WAN because of reduced metadata ops for reading, but it looks like we > incur extra metadata ops for writing. Just a small update based on the most recent patchsets from Trond & Bruce: https://patchwork.kernel.org/project/linux-nfs/list/?series=393567 https://patchwork.kernel.org/project/linux-nfs/list/?series=393561 For the write-through tests, the NFSv3 re-export of a NFSv4.2 server has trimmed an extra GETATTR: Before: originating server <- (vers=4.2) <- reexport server - (vers=3) <- client writing = WRITE,COMMIT,GETATTR .... repeating After: originating server <- (vers=4.2) <- reexport server - (vers=3) <- client writing = WRITE,COMMIT .... repeating I'm assuming this is specifically due to the "EXPORT_OP_NOWCC" patch? All other combinations look the same as before (for write-through). An NFSv4.2 re-export of a NFSv3 server is still the best/ideal in terms of not incurring extra metadata roundtrips when writing. It's great to see this re-export scenario becoming a better supported (and performing) topology; many thanks all. Daire
